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ABSTRACT 


The  Marine  Corps  has  recently  embarked  on  the  development  of  a  Vertical 
Takeoff  and  Landing  Unmanned  Aerial  Vehicle  (VTUAV)  to  replace  the  aging  Pioneer 
system.  This  thesis  examines  the  critical  elements  this  platform  must  possess  in  order  to 
effectively  support  small  units  operating  in  urban  environments.  We  address  this  issue  by 
creating  and  exploring  an  agent-based  simulation  of  a  platoon  conducting  an  urban  patrol 
in  a  setting  similar  to  those  currently  being  encountered  in  Iraq.  The  platoon  utilizes  the 
VTUAV  as  an  intelligence-gathering  asset. 

We  use  an  efficient  designed  experiment  to  generate  data  from  the  simulation 
scenario,  and  then  use  multiple  regression  and  regression  trees  to  relate  the  UAV 
capabilities  to  the  patrol’s  operational  effectiveness.  Our  results  suggest  that  the 
effectiveness  of  a  VTUAV  is  greatly  influenced  by  noise  in  the  urban  warfare 
environment.  We  use  a  loss  function,  along  with  the  regression  models,  to  identify  UAV 
configurations  that  improve  operational  effectiveness  yet  are  robust  to  uncertainties  about 
civilian  and  insurgent  behavior.  The  VTUAV  must  have  high  communication  capability, 
as  well  as  accurate  sensing,  in  order  to  perfonn  well  across  a  range  of  environmental 
conditions. 
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The  reader  is  cautioned  that  computer  programs  developed  in  this  research  may 
not  have  been  exercised  for  all  cases  of  interest.  While  every  effort  has  been  made  within 
the  time  available  to  ensure  that  the  programs  are  free  of  computational  and  logic  errors, 
they  cannot  be  considered  validated.  Any  application  of  these  programs  without 
additional  verification  is  at  the  risk  of  the  user. 
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EXECUTIVE  SUMMARY 


Although  major  combat  in  Iraq  has  ended,  the  task  of  securing  the  nation  is  far 
from  complete.  The  military  forces  in  Iraq  have  the  difficult  task  of  conducting  stability 
and  support  operations  (SASO)  to  root  out  any  insurgent  and  terrorist  activities  in  order 
for  democracy  to  take  hold.  In  order  to  win  the  hearts  and  minds  of  the  people,  and 
stabilize  the  fledgling  democracy  in  Iraq,  the  U.S.  troops  must  focus  on  the  political  and 
population  centers  of  Iraq.  Unfortunately,  it  is  in  this  urban  environment  that  military 
operations  can  exact  a  heavy  toll.  Urban  areas  allow  insurgent  groups  to  inflict  high 
casualties  while  avoiding  decisive  engagements.  Urban  areas  also  offer  a  target-rich 
environment  for  terrorist  tactics,  such  as  the  kidnapping  of  civilian  contractors, 
diplomats,  and  aid  workers,  as  well  as  the  use  of  improvised  explosive  devices  (IEDs). 
Due  to  the  success  of  these  tactics,  the  U.S.  must  find  better  ways  to  utilize  its 
technological  and  firepower  advantages. 

During  the  course  of  the  last  year,  the  Marine  Corps  has  introduced  many 
technological  enhancements  in  order  to  give  our  troops  the  best  possible  advantages.  In 
addition  to  ground  equipment  improvements,  the  Marine  Corps  is  currently  seeking  to 
replace  the  aging  Pioneer  Unmanned  Aerial  Vehicle  (UAV)  with  a  more  capable  platform 
better  suited  for  an  expeditionary  force.  This  new  UAV  will  be  equipped  with  Vertical 
Takeoff  and  Landing  (VTOL)  capability.  With  this  new  capability  the  Marine  Corps  will 
have  greater  flexibility  in  the  areas  that  the  UAV  could  be  employed.  The  new  VTUAV 
will  able  to  span  the  spectrum  from  low  to  high  intensity  combat,  including  urban 
warfare.  Because  of  the  dense  chaos  of  urban  warfare,  we  must  examine  the  capabilities 
that  a  UAV  must  possess  in  order  for  U.S.  forces  to  exploit  this  technological  advantage 
in  such  an  environment. 

In  this  thesis,  we  examine  the  impact  of  providing  a  small  infantry  unit  with  a 
UAV  during  urban  patrolling  operations.  We  create  a  simulation  model  of  this  scenario 
using  an  agent-based  modeling  platform  called  Map  Aware  Non-uniform  Automata 
(MANA).  This  scenario  is  modeled  off  of  the  current  situation  our  forces  are 
encountering  in  Iraqi  urban  areas.  The  platoon  has  one  UAV  that  is  used  as  an 
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intelligence  gathering  asset.  Enemy  forces  operate  in  small  groups  and  utilize  the  cover 
and  concealment  of  the  urban  landscape.  The  insurgent  forces  also  exercise  influence 
over  the  local  populace,  fueling  anti-American  sentiment. 

We  gain  insights  from  this  simulation  by  using  the  technique  known  as  Data 
Fanning.  Data  Farming  involves  generating  a  very  large  number  of  data  points  from 
simple  computer  models,  based  on  a  wide  range  of  model  inputs,  in  order  to  explore  the 
models’  behavior.  These  inputs  include  UAV  capabilities:  its  sweep  width,  observation 
cover,  and  speed,  along  with  communication  reliability,  latency,  and  accuracy. 
Characteristics  of  the  Iraqi  civilians  and  insurgents  are  also  varied  over  broad  ranges.  We 
use  an  efficient  experimental  design  approach  to  specify  the  combination  of  model 
inputs.  Even  so,  over  40,000  simulations  are  run  to  provide  data  for  our  analysis. 

We  use  a  robust  design  approach  to  analyze  the  results.  This  approach  is  based  on 
the  idea  that  a  system  should  be  relatively  insensitive  to  variations  or  noise  in  the 
environment.  Robust  solutions  are  particularly  appealing  for  combat  systems.  In  the 
combat  environment,  military  systems  are  constantly  affected  by  uncontrollable  sources 
of  variation.  This  thesis  is  intended  to  be  a  preliminary  study  of  the  factors  that 
contribute  to  the  success  of  the  VTUAV  in  support  of  a  small  unit  conducting  urban 
operations.  Exact  numerical  results  of  this  study  should  not  be  taken  as  a  literal 
translation  into  real  world  numbers.  It  is  the  insights,  not  the  numbers,  that  are  the  focus 
of  our  investigation. 

Based  on  the  robust  design  philosophy  our  thesis  work  suggests  the  following 

results: 

•  The  hostility  of  the  civilian  population  plays  an  important  role  in  urban 
combat.  If  civilian  hostilities  are  high,  a  more  sophisticated  UAV  might 
be  necessary.  In  environments  with  friendlier  populations,  other  less 
sophisticated  intelligence  assets  might  be  sufficient  to  accomplish  the 
mission. 

•  A  platform  that  has  very  reliable  communication  capability  and  very 
accurate  sensing  capability  is  best  suited  for  the  range  of  environmental 
conditions  modeled.  These  two  factors  were  the  most  dominant  in 
providing  favorable  results. 
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•  The  aggressiveness  of  the  blue  force  has  an  impact  on  the  mean  exchange 
ratio.  In  situations  where  commanders  cannot  tolerate  the  possibility  of 
high  losses,  a  less  aggressive  force  that  relies  more  heavily  on  the  UAV 
for  situational  awareness  and  enemy  engagements  might  be  more 
appropriate. 

•  The  ‘best’  decision  resulting  from  the  robust  design  approach  is  different 
than  the  decision  that  would  have  resulted  by  only  using  mean 
performance.  The  resulting  system  is  more  insensitive  to  uncontrollable 
sources  of  variation  in  the  combat  environment. 

•  By  using  intelligent  design  techniques,  we  are  able  to  capture  the  essence 
of  the  problem  by  using  relatively  few  data  points. 

•  The  data  fanning  environment,  coupled  with  agent-based  models,  provides 
an  excellent  framework  for  gaining  insights  into  interesting  interactions  in 
combat  modeling. 

A  UAV  that  has  reliable  communication  and  accurate  sensing  capability  will 
work  best  across  a  broad  range  of  environments.  If  we  have  some  information  about  the 
environment  such  as  the  civilian  hostility,  we  may  be  able  to  use  a  less  sophisticated 
UAV  and  still  operate  effectively.  Finally,  the  aggressiveness  of  the  blue  forces  plays  a 
key  role  in  achieving  higher  exchange  ratios,  but  it  can  also  result  in  higher  risks.  If  the 
potential  for  high  losses  can  not  be  tolerated,  the  blue  force  should  rely  more  heavily  on 
the  intelligence  gathering  and  targeting  capabilities  of  the  UAV. 
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I.  INTRODUCTION 


Intelligence  is  imperative  to  success  in  urban  warfare.  “ Maneuver 
warfare  requires  a  firm  focus  on  the  enemy”  (MCDP  2,  Intelligence).  Few 
subsequent  tactical  changes  can  overcome  the  far-reaching  impact  of  a 
major  intelligence  error. 


MCWP  3-35.3 

Military  Operations  in  Urban  Terrain  (MOUT) 


A.  MOTIVATION 

In  the  three  years  since  the  attack  on  the  U.S.  on  September  11,  2001,  the  U.S. 
military  has  removed  the  ruling  government  from  two  nations  and  has  begun  the  process 
of  establishing  democratic  governments  in  both.  In  Iraq  the  military  forces  have  the 
difficult  task  of  conducting  stability  and  support  operations  (SASO)  to  root  out  any 
insurgent  and  terrorist  activities  in  order  for  democracy  to  take  hold.  They  also  must  win 
the  support  of  the  people  through  various  humanitarian  projects  and  daily  interaction.  In 
order  to  win  the  hearts  and  minds  of  the  people  and  stabilize  the  fledgling  democracy  in 
Iraq,  the  U.S.  troops  must  focus  on  the  political  and  population  centers  of  Iraq. 
Unfortunately,  it  is  in  this  urban  environment  that  military  operations  can  exact  a  heavy 
toll.  Utilizing  the  advantages  of  the  urban  terrain,  insurgents  have  begun  a  campaign  of 
terror  aimed  at  the  U.S.,  its  coalition  partners,  and  even  Iraqi  citizens.  With  its  superior 
combat  power  and  technological  advantage,  the  U.S.  forces  toppled  Saddam  Hussein’s 
regime  in  just  three  weeks.  However,  stabilization  efforts  are  still  ongoing  over  one  year 
later. 

Urban  areas  allow  insurgent  groups  to  inflict  high  casualties  while  avoiding 
decisive  engagements.  Urban  areas  also  offer  a  target-rich  environment  for  terrorist 
tactics,  such  as  the  kidnapping  of  civilian  contractors,  diplomats,  and  aid  workers,  as  well 
as  the  use  of  improvised  explosive  devices  (IEDs).  Due  to  the  success  of  these  tactics, 
the  U.S.  must  find  better  ways  to  utilize  its  technological  and  firepower  advantages. 
During  the  course  of  the  last  year  the  Marine  Corps  has  introduced  many  technological 
enhancements  in  order  to  give  our  troops  the  best  possible  advantages.  In  addition  to 
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ground  equipment  improvements,  the  Marine  Corps  is  currently  seeking  to  replace  the 
aging  Pioneer  Unmanned  Aerial  Vehicle  (UAV)  with  a  more  capable  platform  better 
suited  for  an  expeditionary  force.  Because  of  the  dense  chaos  of  urban  warfare,  we  must 
examine  the  capabilities  that  a  UAV  must  possess  in  order  for  U.S.  forces  to  exploit  this 
technological  advantage  in  such  an  environment. 

B.  PURPOSE 

The  purpose  of  this  thesis  is  to  utilize  an  agent-based  model  (ABM)  to  develop  a 
simulation  representing  a  small  unit  urban  infantry  patrolling  operation  utilizing  a  UAV. 
The  simulation  will  be  used  to  identify  important  factors,  primarily  related  to  the  UAV, 
that  are  vital  to  mission  success. 

C.  BACKGROUND 

As  part  of  naval  expeditionary  forces  (NEFs),  the  Marine  air-ground  task  forces 
(MAGTFs)  are  forward  deployed  for  rapid  crisis  response.  As  a  rapid  response  force, 
Marines  may  be  employed  in  a  wide  range  of  military  operations.  Studies  have  shown 
that  75%  of  politically  significant  urban  areas,  other  than  allied  or  former  Warsaw  Pact 
territories,  are  located  within  150  miles  of  a  coastline.  As  a  rapidly  deployable  naval 
force  the  Marines  must  be  prepared  to  operate  in  these  large  urban  areas. 

Historically,  the  attacker  usually  wins  in  urban  battles.  This  statistic  may  be 
attributable  to  the  fact  that  urban  warfare  extracts  a  heavy  toll  on  the  attacker,  causing 
him  to  attack  only  when  he  has  a  decisive  advantage — not  only  in  numbers,  but  also  in 
other  factors  such  as  intelligence  and  superior  training  and  equipment. 

Regardless  of  the  size  or  quality  of  defensive  forces,  the  defender  usually  extracts 

large  costs  from  the  attacker  in  time,  resources,  and  casualties.  [MCWP  3-35.3, 

April  1998] 

Military  operations  in  urban  environments  have  historically  involved  small  infantry  units 
and  have  been  casualty  intensive  affairs. 

Current  operations  in  Iraq  demonstrate  the  difficulties  of  patrolling  in  an  urban 
environment.  Insurgents  are  able  to  blend  into  the  local  populace,  containing  many  who 
are  sympathetic  to  their  cause.  Using  IEDs  and  taking  hostages  are  some  of  the  tactics 
being  used  against  U.S.  Forces.  In  Ramadi,  a  city  of  about  400,000  on  the  Euphrates 
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River  west  of  Baghdad,  Marines  are  currently  conducting  security  patrols  looking  for 
weapons  and  insurgents  while  still  trying  to  win  the  hearts  and  minds  of  the  local 
populace.  During  routine  patrols  in  early  April  2004,  the  city  erupted  in  a  series  of 
coordinated  ambushes  by  hundreds  of  Iraqis  shooting  at  Marines  from  alleys,  behind 
windows  and  doors,  and  out  in  the  street. 


When  this  was  taking  place,  two  hours  into  it,  everyone  and  their  mother 
was  shooting  at  us. 


Staff  Sgt.  Damien  Rodriguez 
Regarding  the  fighting  in  Ramadi,  Iraq  April  6,  2004 


It  is  this  type  of  fighting  that  has  served  as  the  basis  for  development  of  the  scenario 
examined  in  this  thesis. 

D.  UNMANNED  AERIAL  VEHICLES  BACKGROUND 

A  Marine  Corps  analysis  on  the  Russian  war  in  Chechnya  stated  that  UAVs  were 
used  extensively  and  effectively  in  the  urban  environment  [FM  3-06.1 1,  2002].  The  use 
of  UAVs  in  military  operations  has  stretched  for  nearly  one  hundred  years.  From  the 
early  years  of  flight,  through  the  Vietnam  War,  to  today,  unmanned  surveillance 
continues  to  play  an  important  role  in  military  operations.  Because  there  are  many 
different  types  of  unmanned  aircraft,  we  use  the  definition  of  a  UAV  from  Joint 
Publication  1-02: 

A  powered,  aerial  vehicle  that  does  not  carry  a  human  operator,  uses 
aerodynamic  forces  to  provide  vehicle  lift,  can  fly  autonomously  or  be 
piloted  remotely,  can  be  expendable  or  recoverable,  and  can  cany  a 
lethal  or  non-lethal  payload.  Ballistic  or  semi  ballistic  vehicles,  cruise 
missiles,  and  artillery  projectiles  are  not  considered  unmanned  aerial 
vehicles. 

With  today’s  technology,  the  UAV  is  far  more  capable  and  sophisticated  than  those 
of  just  a  few  years  ago.  As  technological  advances  continue,  the  UAVs  of  tomorrow  will 
possess  capabilities  that  will  allow  military  commanders  to  receive  large  amounts  of 
relevant  infonnation  more  quickly  and  more  reliably  than  is  possible  using  the  UAVs  of 
today. 
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America’s  first  version  of  a  modem  UAV  was  the  Lightning  Bug.  It  was  used  as  a 
targeting  decoy  during  the  Vietnam  War.  During  the  war  it  was  outfitted  with  primitive 
computers  and  cameras  for  surveillance,  but  limitations  in  its  navigation  system  meant  it 
could  not  be  reliably  deployed.  It  continues  to  hold  the  record  as  the  most  expensive 
drone  ever  built  by  the  U.S.  In  today’s  dollars  it  would  cost  $4.5  billion  [ Spies  That  Fly, 
July  2004]. 

The  next  major  advancement  in  UAVs  occurred  in  1982  during  the  Arab/Israeli 
War  in  southern  Lebanon.  Initially,  the  Israeli  Air  Force  sustained  significant  losses  due 
to  the  Soviet  supplied  anti-air  capability.  In  response  to  these  losses,  the  Israelis 
developed  a  UAV  to  act  as  a  decoy.  This  propeller-driven  UAV  had  no  firepower,  and 
was  controlled  from  the  ground  with  radio  signals.  It  was  slow,  unsophisticated,  and  had 
limited  range,  but  it  carried  signal  generators  that  gave  off  the  radar  signature  of  a  much 
bigger  plane.  By  sending  in  a  wave  of  these  UAVs  before  the  manned  fighters,  the 
Israelis  caused  the  Syrians  to  turn  on  their  radar.  This  allowed  the  Israeli  fighters  to 
launch  anti -radiation  missiles  and  destroy  the  radars.  Later,  the  Israelis  would  equip 
these  UAVs  with  cameras  for  surveillance  operations  [ Spies  That  Fly,  July  2004]. 

Motivated  by  the  success  of  the  Israelis  and  such  acts  as  the  bombing  of  the  Marine 
barracks  in  Lebanon,  the  U.S.  began  to  develop  its  own  UAV  (called  the  Pioneer) 
modeled  after  that  of  Israel.  The  Pioneer  first  saw  action  during  Operation  Desert  Stonn. 
Capable  of  staying  aloft  for  hours,  it  was  assigned  to  battle  ships  for  targeting  purposes. 
It  was  launched  from  the  ship  with  a  rocket-propelled  catapult  and  retrieved  in  a  cargo 
net.  The  Pioneer  enabled  the  U.S.  to  receive  real-time  target  damage  assessment  and 
reduced  the  need  for  human  eyes  on  the  target.  Iraqi  soldiers  actually  tried  to  surrender 
to  these  UAVs  because  they  knew  that  if  they  heard  a  drone  that  bombs  would  usually 
follow  [Spies  That  Fly,  July  2004].  The  Pioneer  had  a  limited  range  due  to  its  line  of 
sight  communications  requirement.  This  challenge  was  soon  answered  with  the 
introduction  of  the  next  U.S.  UAV,  called  the  Predator. 

The  Predator  was  the  first  UAV  that  could  receive  signals  from  satellites  and 
therefore  know  its  position  at  all  times.  It  could  stay  aloft  for  40  hours.  It  first  saw 
service  in  1995  over  Bosnia  [Spies  That  Fly,  July  2004].  It  was  equipped  with  a  camera, 
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infrared  radar  (IR),  Synthetic  Aperture  Radar  (SAR),  and  a  laser  designator  for  targeting. 
The  Predator  is  good  at  keeping  track  of  a  target,  but  its  narrow  field  of  view  limits  its 
ability  to  find  a  target  unless  the  location  is  known. 

The  largest  and  most  sophisticated  U.S.  UAV  is  the  Global  Hawk.  It  was  used  in 
Afghanistan  while  still  in  the  flight  test  stage.  It  is  jet  powered  and  can  cruise  at  an 
altitude  of  65,000  feet  [Spies  That  Fly,  July  2004],  The  Global  Hawk  is  programmed 
from  takeoff  to  touchdown  and  does  not  have  the  capability  of  being  piloted  from  the 
ground.  It  is  a  pure  surveillance  platfonn,  carrying  no  weapons  payload.  It  recently 
completed  a  flight  from  California  to  Australia  without  refueling  [ Spies  That  Fly,  July 
2004],  The  Global  Hawk  is  capable  of  surveying  a  much  larger  area  than  the  Pioneer,  but 
at  $40  million  apiece  it  costs  substantially  more  than  any  other  current  UAV. 

The  Marine  Corps  currently  possesses  two  UAVs  that  are  both  used  for 
surveillance  purposes  only:  the  Pioneer  and  the  Dragon  Eye.  The  Dragon  Eye  is  a  man- 
packed,  propeller-driven  UAV.  It  is  equipped  with  a  small  camera,  has  a  short  range  and 
flying  time,  and  is  used  for  over-the-hill  reconnaissance  only.  The  Marine  Corps  has 
embarked  on  a  program  to  replace  the  Pioneer  with  a  Vertical  Takeoff  and  Landing  UAV 
(VTUAV).  The  VTUAV  is  expected  to  provide  support  to  operations  spanning  the 
spectrum  of  Military  Operations  Other  Than  War  (MOOTW)  to  a  large  intensity  conflict 
[Operational  Requirements  Document  (ORD)  for  the  VTUAV].  Some  of  the  limitations 
of  Pioneer  include  its  limited  shipboard  compatibility,  obsolete  Ground  Control  Stations 
(GCS)  and  inadequate  speed.  The  VTUAV  will  provide  the  capabilities  to  support  the 
Marine  Corps  concepts  of  Operational  Maneuver  From  the  Sea  (OMFTS)  and  Ship  to 
Objective  Maneuver  (STOM).  The  capabilities  of  the  VTUAV  will  include:  target 
acquisition,  surveillance,  reconnaissance,  target  designation,  communications  and  data 
relay,  electronic  warfare,  and  delivery  of  remote  sensors  and  non-lethal  weapons  [ ORD 
for  the  VTUAV].  The  VTUAV  will  be  an  organic  asset  of  the  Marine  Air  Ground  Task 
Force  (MAGTF)  Commander.  Current  fielding  projections  have  the  VTUAV  available  to 
the  FMF  by  FY08  [ORD for  the  VTUAV]. 
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E.  SCOPE 

In  an  urban  combat  scenario  it  is  possible  to  examine  hundreds  of  factors  that 
might  contribute  to  mission  success.  The  introduction  of  an  intelligence-gathering  asset 
such  as  a  UAV  to  this  environment  adds  to  the  complexity  of  the  analysis.  In  order  to 
conduct  a  practical  examination  of  issues  pertinent  to  the  development  and  use  of  the 
VTUAV  in  conjunction  with  small  unit  urban  operations,  this  thesis  will  adhere  to  the 
following  plan  for  analysis: 

•  Review  urban  warfare  tactics. 

•  Understand  potential  enemy  tactics  and  courses  of  action. 

•  Identify  measures  of  effectiveness. 

•  Explain  the  capabilities  and  limitations  of  the  modeling  tool. 

•  Explain  the  development  of  the  simulation  scenario. 

•  Discuss  data  farming  and  its  application  to  the  urban  patrolling  model. 

•  Analyze  the  data  from  our  simulation  runs  and  fit  models  to  the  output. 

•  Examine  the  results  of  our  analysis  as  it  pertains  to  urban  patrolling  with 
the  VTUAV. 

F.  AGENT-BASED  MODELS 

This  thesis  will  gain  insights  to  questions  by  using  the  technique  known  as  Data 
Fanning.  Data  Fanning  is  a  method  that  generates  a  very  large  number  of  data  points 
from  simple  computer  models,  based  on  a  wide  range  of  model  inputs.  The  analyzed 
results  can  yield  surprises,  a  better  understanding  of  the  model  perfonnance,  and 
interesting  outcomes.  The  Marine  Corps  Warfighting  Laboratory’s  Project  Albert  has 
provided  the  necessary  infrastructure  for  this  research  capability  to  the  Marine  Corps. 

Project  Albert’s  Data  Farming  approach  is  achieved  through  the  use  of  agent- 
based  models  or  distillations.  Agent-based  models  have  entities  that  are  controlled  by 
decision-making  algorithms,  enabling  the  analyst  to  examine  how  agents  interact  with 
one  another  in  potentially  interesting  and  non-linear  ways.  These  are  simple  models  that, 
when  coupled  with  high  performance  computing,  can  generate  potentially  millions  of 
data  points  for  analysis.  The  agent-based  modeling  platfonn  used  in  this  thesis  is  Map 
Aware  Non-uniform  Automata  (MANA).  MANA  was  developed  by  the  New  Zealand 
Defence  Technology  Agency  and  was  inspired  by  the  work  of  Andy  Ilanchinski  and 
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others  using  agent  based  models  [Galligan  et  al.,2004].  MANA  examines  the  interactions 
among  agents  that  have  their  own  personality  traits  and  are  able  to  move  autonomously. 
The  parameters  for  the  agents  fall  into  four  basic  types:  (1)  personality,  (2)  movement, 
(3)  capabilities,  and  (4)  movement  constraints  [Galligan  et  ah,  2004]. 

By  creating  agents  in  MANA  and  varying  these  four  types  of  parameters  we  can 
examine  a  wide  range  of  behaviors.  Of  particular  interest  are  the  levels  of  several 
parameters  or  factors  that  potentially  lead  to  interesting  and/or  unexpected  results. 
Additionally,  the  way  agents  behave  with  one  another,  towards  the  enemy,  with  civilians 
and  the  VTUAV  could  also  produce  interesting  insight  into  the  scenario. 

Once  the  scenario  has  been  created  in  MANA  and  data  have  been  generated,  we 
can  apply  regression  and  other  data  analysis  techniques  to  gain  a  better  understanding  of 
the  interactions  between  the  factors  and  provide  a  glimpse  of  the  potential  trade  space  for 
decision  makers. 
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II.  MODEL  DESCRIPTION  AND  DEVELOPMENT 


A.  SOME  DOCTRINAL  PRINCIPLES  OE  MOUT 

The  study  of  modern  urban  battles  has  resulted  in  some  key  insights  into  the 
major  factors  of  urban  combat.  MCWP  3-35.3  [April,  1998]  lists  a  few  of  these  insights 
as  the  following: 

•  MOUT  is  infantry  intensive; 

•  A  tactical  battle  may  have  far-reaching  implications,  with  operational  and 
possibly  strategic  repercussions; 

•  Intelligence  is  imperative  to  success; 

•  Surprise  is  a  combat  multiplier; 

•  As  force  ratio  increases  in  favor  of  the  attacker,  combat  duration 
decreases; 

•  Urban  warfare  is  time  consuming;  and 

•  Attack  of  an  urban  area  is  costly  to  the  attacker  in  terms  of  resources  and 
casualties. 

Because  of  the  nature  of  urban  terrain,  units  often  become  separated  and  isolated. 
This  forces  the  lighting  to  become  a  series  of  small-unit  actions.  When  fighting 
unconventional  forces  in  MOUT,  our  forces  will  usually  be  operating  under  restrictive 
rules  of  engagement  (ROE).  Insurgent  forces  will  exploit  our  ROE  and  use  the  local 
population  to  their  advantage  when  developing  their  defensive  plan. 

The  preferred  method  of  advancing  along  city  streets  is  the  double  column.  This 
allows  360  degrees  of  security  and  mutual  support  as  shown  in  Figure  1. 
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Figure  1.  Patrolling  Along  an  Urban  Street  [from  MCWP  3-35.3,  April  1998]. 


During  a  patrol,  enemy  contact  is  possible  even  if  an  area  is  deemed  secure.  If  an 
enemy  is  engaged,  the  patrol’s  preferred  choice  should  be  to  return  fire  immediately  and 
conduct  a  hasty  clearing  of  an  adjacent  structure  in  order  to  seek  cover.  This  allows  the 


Marines  to  better  acquire  the  enemy’s  position  and  to  deliver  well-aimed  shots  in  order  to 
use  their  firepower  more  effectively. 

The  nature  of  combat  in  urban  areas  affects  the  employment  of  weapon  systems. 
Engagement  areas  are  usually  close  in  MOUT.  According  to  MCWP  3-31, 

Studies  and  historical  analyses  have  shown  that  only  5  percent  of  all 
targets  are  more  than  100  meters  away.  About  90  percent  of  all  targets 
are  located  50  meters  or  less  from  the  identifying  Marine.  Few  personnel 
targets  will  be  visible  beyond  50  meters,  and  most  occur  at  35  meters  are 
less. 

The  M16  rifle  and  M249  Squad  Automatic  Weapon  (SAW)  are  the  most  frequently  used 
weapons  in  urban  areas  primarily  because  these  make  up  the  chief  firepower  of  the 
infantry  platoon,  but  also  because  strict  ROE  and  the  fear  of  collateral  damage  prevent 
the  use  of  indirect  fire. 

B.  SMALL  UNIT  URBAN  OPERATIONS  SCENARIO 

This  analysis  focuses  on  an  infantry  platoon  conducting  patrolling  operations  in 
an  urban  environment.  The  situation  is  modeled  off  of  the  current  situation  our  forces  are 
encountering  in  Iraqi  urban  areas.  The  mission  of  the  patrol  is  to  neutralize  insurgents 
who  are  operating  in  the  area,  as  well  as  to  provide  a  presence  to  the  local  populace.  The 
platoon  consists  of  39  members  armed  with  small  arms.  They  are  supported  by  one  UAV 
providing  intelligence  updates  on  possible  enemy  locations  that  is  also  capable  of 
providing  observational  cover  if  the  enemy  engages  the  platoon.  The  situation  in  the  city 
is  tense  and  the  citizens  are  not  happy  with  the  presence  of  U.S.  troops.  Many  citizens 
are  sympathetic  to  the  insurgents  and  will  aid  them  in  certain  situations.  By  using  the 
urban  terrain  to  their  advantage,  the  insurgents  operate  in  small  units  and  avoid  large- 
scale  engagements.  Using  the  cover  and  concealment  afforded  to  them  by  the  buildings, 
they  will  usually  only  initiate  the  engagement  when  they  have  the  advantage  of  surprise. 
The  insurgents  operate  as  loosely  organized  irregular  units  with  the  ability  to  blend  in 
with  the  local  population,  but  because  of  their  desire  to  make  a  political  statement  and 
influence  the  local  citizens,  they  also  have  a  desire  to  make  their  presence  known. 
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C.  MANA 

All  models  are  wrong,  but  some  are  useful. 

George  Box 

As  explained  in  Chapter  I,  we  chose  MANA  as  the  platfonn  for  exploring  this 
scenario.  Much  of  the  infonnation  that  follows  is  taken  from  the  April  2004  version 
3.0.35  user’s  manual  [Galligan  et  al.,  2004].  MANA  was  developed  because  of  a 
frustration  with  other  more  complex,  physics  based  models  that  were  available.  While 
highly  detailed,  these  models  were  unable  to  capture  other  intangible  factors  of  combat — 
such  as  human  behavior — which  can  be  critical  determinants  of  battle  outcomes.  The 
developers  offer  this  advice  for  those  using  MANA: 

There  must  be  a  clear  idea  of  which  aspect  of  warfare  the  scenario  is 
addressing,  and  what  the  entities  are  trying  to  do.  Though  such  an 
approach  may  seem  pre-potted,  the  non-linear  nature  of  the  model  ensures 
that,  regardless  of  the  modeller’s  preconception,  a  startlingly  large 
number  of  outcomes  are  possible.  Such  a  range  of  outcomes  is 
characteristic  of  complex  adaptive  systems,  and  occurs  even  with  quite 
simple  rules  of  behaviour  [Galligan  et  al,  2004] . 

It  is  this  range  of  possible  outcomes  that  is  of  interest  to  decision  makers.  With  MANA’s 
user’s  manual  and  Graphical  User  Interface  (GUI)  it  is  relatively  simple  to  build  a 
scenario  in  a  short  amount  of  time. 

D.  MODELING  OUR  SCENARIO  IN  MANA 

This  section  focuses  on  how  our  scenario’s  battlefield  and  agents  were  created  in 
MANA.  Screen  shots  are  shown  to  aid  the  reader  in  understanding  model  development 
process. 

1.  Battlefield  Development 

The  MANA  battlefield  is  made  up  of  grids  that  can  range  from  50  to  1000  pixels 
(or  cells)  on  a  side.  The  default  terrain  is  a  200  x  200  grid.  Our  model  uses  a  1000  x 
1000  design,  which  allows  for  the  highest  resolution.  Each  cell  can  be  occupied  by  a 
single  live  entity.  The  map  file  is  based  on  a  standard  Windows  bitmap  and  different 
colors  have  an  affect  on  an  agents’  ability  to  shoot,  move,  or  communicate.  MANA  has 
preset  terrains  for  its  default  battlefield,  but  the  user  can  create  additional  terrains  to 
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provide  cover  and  concealment,  impede  or  facilitate  movement  and  communication,  or 
act  as  barriers.  Figure  2  shows  the  Scenario  Map  Editor  where  changes  can  be  made  to 
the  terrain. 


Figure  2.  Scenario  Editor. 


2.  Creating  MANA  Agents 

In  MANA  a  group  of  agents  of  any  size  (1-1000)  is  known  as  a  squad.  These 
squads  have  the  same  properties  and  can  change  between  states  as  a  group  or 
individually.  States  are  a  set  of  values  that  determine  the  agents’  behavior.  This  ability 
of  agents  to  change  their  behavior  based  on  a  stimulus  is  a  key  component  to  modeling 
combat  behavior.  The  squads  created  for  this  scenario  are  listed  in  Table  1. 
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Squad  = 

Squad  Name 

Allegiance 

#  of  Agents 

Description 

Icon 

Blue  Squad  1 

Blue 

19 

Infantry 

w,M16 

Blue 

Soldier 

UAV 

Blue 

j 

UAV 

Blue 

Airplane 

- 

Chilians 

Neutral 

90 

Unhappy  with 
U.S. 

Occupation 

Yellow- 

Person 

HD 

Red 

Improvised 

Explosive 

Device 

Red 

Cross 

5 

ZED  Observer 

Red 

Needed  to 
detonate  IED 

Red 

Soldier  in 

Prone 

6 

Red  Squad  1 

Red 

- 

Insurgents 
\v  AK  47 

Red 

Soldier 

1 

Red  Squad  2 

Red 

Insurgents 
v  AK  47 

Red 

Soldier 

S 

Red  Squad  5 

Red 

_* 

Insurgents 
w,  AK  47 

Red 

Soldier 

9 

Red  Squad  4 

Red 

_* 

Insurgents 
w  AK  47 

Red 

Soldier 

10 

Red  Squad  5 

Red 

Insurgents 
w,  AK  47 

Red 

Soldier 

Red  Squad  6 

Red 

Insurgents 
w  AK  47 

Red 

Soldier 

12 

Blue  Squad  2 

Blue 

19 

Infantry 

w/M16 

Blue 

Soldier 

13 

Red  Squad  7 

Red 

_* 

Insurgents 
w  AK  47 

Red 

Soldier 

14 

Red  Squad  S 

Red 

j 

Insurgents 
w  AK  47 

Red 

Soldier 

15 

Red  Squad  9 

Red 

_* 

Insurgents 
\v  AK  47 

Red 

Soldier 

Table  1.  Urban  Patrol  Squads 


Squads  can  be  guided  through  the  scenario  by  a  series  of  waypoints  designated  by  the 
user.  Their  propensity  to  go  to  a  waypoint  is  determined  by  the  user  via  the  personality 
settings  of  each  squad,  which  we  describe  shortly. 

The  squad’s  organic  situational  awareness  (SA)  and  inorganic  SA  are  important 
aspects  of  modeling  information  and  communication  on  the  battlefield.  Whenever  a 
squad  member  sees  or  detects  another  entity  on  the  battlefield,  this  detection  is 
transmitted  to  other  members  of  the  squad  and  stored  in  memory  for  a  user-specified 
number  of  time  steps.  This  allows  the  squad  to  have  a  collective  picture  of  the  battlefield 
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and  make  decisions  based  on  their  personality  weightings.  Important  to  this  scenario  are 
the  squads’  inorganic  SA.  Inorganic  SA  is  information  passed  to  one  squad  by  another 
squad.  Like  the  squad’s  organic  SA,  the  inorganic  SA  is  stored  in  memory  for  a  user- 
specified  number  of  time  steps.  This  feature  allows  the  user  to  determine  the  importance 
of  the  age  of  information  when  making  decisions.  In  the  Inorganic  SA  Tab  (Figure  3)  we 
see  that  the  user  has  the  ability  to  vary  certain  factors  that  pertain  to  the  communication 
aspects  of  passing  information  over  a  communication  net.  Accuracy,  reliability,  and 
latency  are  just  of  few  of  the  parameters  than  can  be  varied. 


Figure  3.  Inorganic  SA  Communication  Properties. 

A  squad’s  current  organic  (internally  generated)  or  inorganic  SA  can  be  viewed  at 
anytime  by  selecting  the  appropriate  SA  from  the  view  menu,  as  seen  in  Figure  4.  Figure 
4  displays  the  information  being  passed  to  the  squad  via  the  communications  link  of  other 
designated  squads.  In  our  scenario  the  inorganic  SA  that  the  blue  squad  is  receiving  is 
from  the  UAV.  Different  colored  boxes  are  displayed  around  different  red  icons.  The 
yellow  boxes  indicate  the  UAV  has  detected  a  hostile  civilian  and  the  grey  boxes  indicate 
an  insurgent  detection.  This  information  is  passed  to  Squad  1  and  displayed  in  Figure  4. 
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Figure  4.  Squad  Inorganic  SA  (Best  Viewed  in  Color). 

3.  The  Urban  Patrol  Scenario 

Figure  5  displays  a  screenshot  of  the  initial  condition  of  the  scenario  for  this 
analysis.  The  terrain  is  an  urban  environment  with  buildings,  roads,  alleys  and  structures. 
This  map  is  actually  modeled  off  of  the  MOUT  training  facility  at  Ft.  Polk,  Louisiana. 
The  colors  of  the  map  represent  various  structures  and  each  has  unique  cover  and 
concealment  attributes  as  well  as  an  effect  on  squad  movement.  Yellow  represents  a 
major  street.  It  offers  ease  of  movement  but  no  cover  and  concealment.  Grey  areas  are 
walls  or  other  solid  structures.  These  areas  offer  good  cover  but  entities  are  unable  to 
pass  through  or  over  them.  The  light  green  areas  represent  the  interior  of  buildings. 
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These  areas  offer  good  concealment  and  restrict  movement.  The  black  area  of  the  map  is 
representative  of  off-road  urban  terrain  such  as  alleys,  sidewalks  and  small  streets.  These 
areas  offer  little  cover  and  concealment,  while  movement  is  only  slightly  more  restrictive 
than  that  of  major  streets. 


MANA  -  3BlueSquads 


[f  Pause 

►  Run 

2?  MultiRun 

M  Reset 

Step  Del.iy[ms] 

P  I 

Max.  Steps 

fiofio 

Seed 

Lock  r  |-548014901 


Clt  Log  |  Save  Log  | 

Width:  1000  Height:  1000 
Sound  Flags  LOS  BG  Int  Fire 
Net  Dead  Anim  Path 

STOPPED 

RecAgts  RecSteps  RecDets 
ReclocsRecMDets  RecComms 
RecPos  RecRedDets  RedstEn 


Run:  1 

Real  (x,y)=0.0,  84.5  j 


Step: 0 

Model  (x,y)=0,  424 


Current  Run:  Blue  Cas:  0 

Terrain=  Urban  Offroad,  Elevation=  0.0m 


Red  Cas:  0 


Figure  5.  Urban  Patrol  Scenario  (Best  Viewed  in  Color). 


The  blue  patrol  starts  at  the  bottom  center  of  the  map  and  proceeds  north  through 
the  city  following  a  series  of  waypoints.  The  blue  platoon  consists  of  two  squads  totaling 
39  agents.  The  blue  UAV  will  provide  intelligence  to  the  patrol  by  conducting  a  search 
of  the  city  utilizing  an  alternating  lawnmower  search  pattern.  The  UAV  can  be 
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dynamically  re-tasked,  represented  by  a  state  change,  to  provide  observation  cover  should 
the  patrol  encounter  enemy  agents  along  the  patrol  route.  Following  doctrine,  the  two 
squads  move  in  parallel  to  one  another  along  the  outer  edges  of  the  street.  Barring  any 
enemy  contact  the  platoon  has  a  propensity  to  simply  move  from  waypoint  to  waypoint. 
Once  the  squad  makes  contact  with  the  enemy  it  will  first  seek  cover  and  then  proceed  to 
engage  the  red  agents,  again  following  MOUT  doctrine.  If  the  patrol  is  not  engaged  with 
the  enemy  and  the  UAV  detects  and  transmits  enemy  contact  to  the  patrol,  then  the  patrol 
will  have  a  propensity  to  move  towards  the  UAV  contact  to  engage. 

There  are  ten  squads  of  insurgents,  represented  on  the  map  as  red  soldiers:  nine 
squads  of  three  agents  each,  and  one  squad  with  one  agent.  The  insurgents  have  a 
tendency  to  remain  inside  buildings,  where  they  are  difficult  for  the  UAV  to  detect,  until 
they  see  the  blue  patrol.  Insurgents  occupying  the  interior  of  a  building  or  engaging  blue 
forces  from  windows  have  a  lower  probability  of  detection  than  those  engaging  in  the 
open.  This  lower  probability  of  detection  is  represented  in  MANA  by  higher  stealth 
settings  for  the  red  agents.  Once  the  red  agents  spot  the  blue  patrol,  they  then  begin  to 
move  and  engage,  resulting  in  decreased  stealth  and  increased  probability  of  detection. 
The  enemy  IED  is  represented  by  a  red  cross.  An  observer,  represented  by  a  red  soldier 
in  the  prone  position,  controls  the  IED.  If  this  observer  notices  that  blue  soldiers  are 
within  range  of  the  IED,  he  will  arm  the  device.  If  the  observer  is  killed  prior  to  arming, 
the  device  is  disabled.  The  IED  can  only  be  armed  if  the  observer  has  a  clear  line  of  site 
to  the  device.  In  order  to  represent  randomness  in  the  IED’s  reliability,  the  observer’s 
position  is  randomly  generated  so  that  it  will  not  have  a  clear  line  of  site  to  the  IED  for 
each  run. 

Neutrals  are  represented  as  yellow  figures  on  the  map.  Their  initial  placements 
and  movements  are  both  random.  If  they  come  in  close  contact  with  an  insurgent,  they 
may  undergo  a  state  change  and  become  hostile  to  the  blue  forces  for  a  period  of  time. 
This  state  change  is  represented  on  the  map  by  the  figure  changing  colors  from  yellow  to 
red. 

4.  Squad  Personalities 

The  Squad  Personalities  screen,  shown  in  Figure  6,  is  the  method  MANA  uses  for 

establishing  an  agent’s  characteristics  for  certain  types  of  actions.  The  various 
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weightings  given  under  agent,  squad  or  inorganic  SA  affect  the  squads’  propensity 
towards  certain  actions.  These  ordinal  weightings  can  vary  from  -100  to  100.  For 
example,  an  agent  with  a  weighting  of  100  for  Enemies  will  have  a  strong  desire  to  move 
toward  enemy  agents,  while  an  agent  with  a  weighting  of -100  will  have  a  strong  desire 
to  move  away  from  enemy  agents.  It  is  worth  mentioning  that  it  is  not  necessary  for  an 
entity  to  have  a  weighting  of  100  to  have  a  strong  desire  to  move  toward  the  enemy.  As 
long  as  the  Enemies  weighting  is  positive  and  greater  than  any  of  the  other  personality 
weighting,  the  entity’s  strongest  desire  will  be  to  move  toward  enemy  agents.  In  Figure  6 
we  see  the  personalities  of  the  blue  platoon  in  its  default  state  (indicated  by  the 
highlighted  box  at  the  top  of  the  Trigger  State  window  shown  on  the  right-hand-side  of 
the  screen).  The  greatest  influence  on  the  agents  is  the  detection  of  an  enemy  agent  via 
the  blue  agent  or  blue  squad  organic  SA.  This  propensity  to  engage  the  red  agents  is  in 
keeping  with  the  mission  of  the  patrol.  Because  the  weighting  of  the  squad  SA  is  greater 
than  inorganic  SA,  the  squad  will  not  break  contact  with  an  enemy  to  seek  other  enemy 
reported  elsewhere  by  the  UAV.  If  the  patrol  is  not  engaged  and  receives  inorganic  SA 
from  the  UAV,  it  will  have  a  propensity  to  move  towards  the  inorganic  enemy  contact. 
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Edit  Squad  Properties 
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Figure  6.  Personality  Settings  of  Squad  1  in  Default  State. 


Checking  a  trigger  state  in  the  Trigger  State  window  allows  the  user  to  change  the 
personality  weightings,  or  other  parameters  in  other  tabs,  of  the  agents  based  on  certain 
actions  of  other  agents  or  how  an  agent  is  passed  information.  The  trigger  state  is  given  a 
user  defined  duration  and  fall-back  state  once  the  specified  time  in  the  trigger  state  has 
elapsed.  Both  these  are  shown  at  the  bottom  of  the  Trigger  State  window.  For  example, 
the  state  change  in  Figure  7  is  for  the  blue  patrol.  Upon  enemy  contact,  the  patrol  will 
have  a  propensity  to  seek  cover  for  a  duration  of  100  time  steps,  and  then  return  to  its 
default  personality.  This  represents  the  doctrinal  tactic  of  seeking  cover,  returning  fire, 
and  then  proceeding  to  clear  a  building. 
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Figure  7.  Duration  and  Fallback  for  the  Squad  En  Contact  Trigger  State. 

5.  Ranges  Tab 

Figure  8  displays  the  Ranges  Tab  for  the  blue  platoon.  The  characteristics  in  this 
tab  can  change  based  on  the  state  of  the  squad.  In  the  General  window  we  can  designate 
an  icon  for  the  squad,  give  it  an  allegiance,  assign  it  a  threat  for  the  enemy’s  SA,  give  it 
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an  agent  class  so  that  the  enemy  can  detennine  what  weapon  to  use  against  it,  and  choose 
the  squad’s  movement  speed.  Our  blue  platoon,  like  other  foot-mobile  agents,  has  a 
movement  speed  of  1,  which  means  it  can  move  one  pixel  in  one  time  step. 


Figure  8.  Squad  Ranges  Tab. 


The  Enemy  Interaction  window  designates  how  many  hits  it  will  take  to  kill  the 
agent,  the  agent’s  stealth,  and  the  armor  thickness.  The  stealth  value  is  the  percent  of 
time  (0-100)  that  the  agent  will  not  be  detectable.  For  our  scenario,  the  blue’s  stealth  is 
very  low  due  to  its  desire  to  show  a  presence.  The  red  stealth  is  much  higher  but  will  be 
reduced  if  it  engages  the  blue  forces. 

The  Misc  window  designates  how  close  an  entity  must  be  to  a  waypoint  in  order 
to  consider  it  reached.  A  value  of  0  means  that  each  entity  would  have  to  be  exactly  on 
top  of  the  waypoint  in  order  to  consider  it  reached.  This  would  result  in  the  squad  being 
channeled  into  a  particular  area,  which  would  decrease  dispersion  and  increase  the 
chances  of  casualties.  For  the  purpose  of  our  scenario,  the  agent  must  be  within  50 
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meters  to  consider  the  waypoint  reached.  The  Sensor  Capabilities  window  for  an  agent  is 
used  to  specify  the  range  over  which  it  can  detect  and  classify  another  agent. 
Classification  ranges  can  be  further  broken  down  into  probabilities  at  different  ranges. 
For  our  scenario,  we  have  locked  the  detection  range  to  the  classification  range  and  set 
the  range  to  50  for  our  blue  platoon.  This  distance  is  in  keeping  with  MCWP  3.35.3.  The 
Fuel  window  is  used  for  agents  needing  refueling.  It  was  not  used  in  this  scenario. 

6.  Weapons  Tab 

The  Weapons  Tab  is  shown  in  Figure  9.  The  Weapons  Tab  allows  the  user  to 
define  how  many  weapons  the  agents  can  use,  as  well  as  the  weapon  types  and 
accuracies.  In  this  scenario  the  blue  agents  have  greater  kill  probabilities  than  red  agents 
to  reflect  better  weapons  and  training.  The  weapon  type  used  by  all  agents,  other  than  the 
IED,  is  a  kinetic  energy  or  small  arms  weapon.  Blue  agents  also  have  a  second  weapon 
type  that  allows  them  to  engage  targets  using  the  inorganic  SA  provided  to  them  from  the 
UAV.  Because  our  forces  will  normally  be  operating  under  strict  ROE  in  this  type  of 
scenario,  the  weapon  used  based  on  inorganic  SA  is  also  a  small  arms  direct  fire  weapon, 
rather  than  a  high  explosive  round,  such  as  artillery  or  grenade  launchers.  The  IED  is  a 
high  explosive  devise  with  a  kill  radius.  Ranges  for  all  the  weapons  in  this  scenario  are 
based  on  MCWP  3-35.3. 
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Figure  9.  Weapons  Tab. 


7.  Movement 

Figure  10  depicts  the  Algorithm  Tab  that  controls  the  movements  of  the  agents. 
The  Stephen  Algorithm  is  the  default  setting  and  is  used  for  this  scenario.  This  algorithm 
considers  all  moves  within  the  range  of  the  agent,  and  then  chooses  a  move  based  on  the 
agent’s  personality  weighting  and  movement  constraints.  The  Precision  Move  Selection 
determines  the  randomness  in  an  agent’s  movement.  A  low  setting  corresponds  to  little 
randomness,  which  is  the  setting  for  blue  agents  in  our  scenario.  Red  agents  have  a 
higher  setting,  giving  them  a  greater  randomness  in  their  movement. 
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Figure  10.  Movement  Algorithm  Tab. 

Under  General  Movement  Settings  the  user  can  further  define  movements.  For 
instance,  if  the  Going  affects  speed  option  is  checked,  then  the  type  of  terrain  will  affect 
the  agent’s  speed  and  ability  to  navigate  obstacles.  This  option  is  checked  for  all  agents 
in  our  scenario  except  for  the  UAV,  which  flies  over  structures  on  the  ground. 

This  concludes  our  discussion  of  model  development.  We  have  seen  that  MANA 
is  a  simple  tool  that  allows  the  user  to  create  detailed  scenarios  in  a  fairly  short  amount  of 
time.  The  model  developed  was  specifically  for  a  small  unit  conducting  patrolling 
operations  in  an  urban  environment  with  a  UAV.  From  the  discussion  it  is  not  difficult  to 
imagine  countless  different  scenarios  with  different  parameters  that  could  be  examined 
using  this  tool. 
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III.  ANALYSIS  METHODOLOGY 


The  Data  Farming  environment  of  MANA  is  excellent  for  exploring  the  effects 
and  interactions  of  many  factors  over  many  levels.  MANA  coupled  with  high 
performance  computing  provides  decision  makers  with  potentially  interesting  outcomes 
based  on  the  interactions  of  the  various  factors  being  modeled.  With  many  factors  being 
examined  over  many  levels,  it  is  essential  that  experimental  designs  be  chosen  that  can 
offer  an  intelligent  way  to  sample  over  a  wide  range  of  factors  and  levels. 

This  chapter  explains  the  methodology  for  analyzing  the  urban  patrol  scenario. 
We  will  discuss  the  robust  design  process  that  will  be  used  for  designing  and  analyzing 
our  experiment.  We  will  discuss  the  measures  of  effectiveness  (MOEs)  that  were  chosen 
for  this  scenario  and  the  statistical  methods  for  analyzing  the  data  and  factors  important 
to  the  success  of  the  model.  Also  covered  in  this  chapter  is  the  data  collection  plan 
chosen  for  exploring  our  MANA  scenario. 

A.  ROBUST  DESIGN 

The  process  used  to  determine  the  best  design  for  this  scenario  is  known  as  robust 
design.  This  approach  is  an  optimization  process  that  espouses  that  a  design’s  mean 
performance  should  not  be  the  only  criteria  for  the  best  design — a  good  system  design 
should  also  be  relatively  insensitive  to  uncontrollable  factors  in  the  environment. 
Genichi  Taguchi,  who  used  this  process  in  the  manufacturing  field  for  engineering 
product  design,  pioneered  this  approach. 

He  found  that  it  was  often  more  costly  to  control  causes  of  manufacturing 
variation  than  to  make  a  process  insensitive  to  these  variations,  and 
through  the  use  of  simple  experimental  designs  and  loss  functions  was 
often  able  to  greatly  improve  product  perfonnance  by  ‘building  in’  the 
quality.  [Sanchez  et  ah,  1996] 

Although  originally  used  in  the  manufacturing  field,  a  design  insensitive  to 
uncontrollable  factors  is  particularly  appealing  for  military  systems.  In  the  combat 
environment,  military  systems  are  constantly  affected  by  uncontrollable  sources  of 
variation.  The  framework  for  a  robust  design  that  follows  is  detailed  in  [Sanchez  et  ah, 
1996]:  (1)  select  the  performance  measures;  (2)  specify  a  loss  function;  (3)  identify  the 
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factors;  (4)  plan  the  experiment;  (6)  analyze  the  results;  (7)  refine  the  metamodels;  (8) 
select  the  best  process  design.  In  the  following  sections  we  examine  each  step. 

B.  MEASURES  OF  EFFECTIVENESS  (MOE) 

An  MOE  is  a  quantitative  measure  of  performance.  In  our  scenario  we  have 
chosen  the  red:blue  kill  ratio,  also  known  as  the  exchange  ratio  (ER).  Rather  than  just 
look  at  red  kills,  we  would  like  a  performance  measure  that  takes  into  account  blue 
casualties.  Although  we  expect  blue  casualty  rates  to  be  high  in  this  scenario,  we  would 
like  to  find  a  combination  of  factors  that  would  make  the  ratio  as  high  as  possible.  For  a 
robust  design  it  is  helpful  to  have  an  associated  target  value  x,  which  represents  the  ideal 
exchange  ratio.  In  a  perfect  world  there  would  be  no  blue  casualties;  however,  we  know 
that  this  is  virtually  impossible  in  urban  warfare.  Actual  exchange  ratios  have  varied 
widely  in  urban  warfare.  For  instance,  the  exchange  ratio  for  the  battle  of  Hue  City 
during  the  Vietnam  War  was  1.6,  while  the  exchange  ratio  for  the  attacking  Iraqis  during 
the  battle  for  Khorrasmshahr  during  the  Iraq/Iran  War  was  0.76  [Lamont,  1999].  Because 
the  battle  for  Hue  was  seen  as  a  tactical  victory  we  will  use  the  optimistic  value  of  x  =  1.5 
as  our  target. 

C.  THE  LOSS  FUNCTION 

The  next  step  in  the  robust  design  process  is  to  define  a  loss  function  that  will 
convert  the  performance  measure  into  dimensionless  units.  This  function  measures  the 
cost  associated  with  the  variation  of  the  system  relative  to  x.  If  the  system  was  ideal,  our 
MOE  would  be  equal  to  x  and  the  variance  would  be  zero.  However,  this  is  highly 
unlikely  to  occur,  so  some  way  of  defining  and  measuring  loss  is  necessary.  For  our 
design  we  will  utilize  a  quadratic  loss  function.  Letting  x  and  Y(x)  represent  a  vector  of 

decision  factors  and  the  associated  MOE,  respectively,  and  letting  Q  be  the  noise  factors, 
then  the  quadratic  loss  function  can  be  written  as: 

1(x)  =  c[Y(x)-t]2  (1) 

where  the  scaling  constant  c  is  used  to  convert  losses  into  monetary  units.  For  our 
purposes  the  scaling  constant  will  be  one.  From  equation  (1)  the  expected  loss  is  defined 
as: 
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£(/(x))  =  c[o2+(|Ix-t)2]  (2) 

where  o2  is  the  variance  and  |lx  is  the  mean  associated  with  the  vector  of  decision 
factors  x;  see  [Sanchez  et  al.  1996]  for  details.  Because  the  variability  is  not  constant 
across  different  designs,  the  loss  function  in  equation  (2)  is  a  better  measure  of  a  ‘good’ 
system  than  just  considering  the  difference  between  the  mean  of  the  MOE  and  the  target 
value. 

Although  we  are  using  a  quadratic  loss  function  for  this  study,  it  is  by  no  means 
the  only  loss  function  appropriate  for  designing  robust  systems.  We  have  chosen  the 
quadratic  loss  functions  for  several  reasons.  The  first  reason  is  that  the  values  are  easily 
interpreted  to  real  world  tenns — the  square  root  of  the  loss,  like  the  standard  deviation,  is 
in  the  original  units  of  the  problem.  Second,  we  know  from  examining  our  data  that  we 
will  not  encounter  values  with  the  means  greater  than  1.5.  This  is  important  because, 
assuming  that  high  exchange  ratios  are  better,  the  quadratic  loss  function  would  associate 
large  losses  with  any  exchange  ratios  much  higher  than  our  target  value.  With  this 
potential  problem  in  mind,  another  possible  loss  function  is  the  following: 

l{x)  =  -  log(7 (x))  =  -log(exchange  ratio)  (3) 

The  expected  loss  can  then  be  estimated  by  the  average  of  the  -log(exchange  ratio) 
values  for  all  runs  and  replications  at  design  point  x.  With  this  loss  function  we  are  not 
penalized  for  achieving  exchange  ratios  higher  than  a  specified  target  value — this  loss 
function  does  not  even  require  the  analyst  to  specify  a  target  value.  One  disadvantage  of 
using  -log(exchange  ratio)  is  that  all  observed  exchange  ratios  must  be  non-zero.  It  is 
also  not  as  natural  a  scale  for  people  to  use  to  compare  and  contrast  alternatives.  In  our 
analysis  we  will  consider  the  results  of  the  loss  function  in  equation  (1)  with  a  target 
value  of  1.5,  show  what  happens  if  a  slightly  different  target  value  is  used,  and  compare 
with  those  using  the  loss  function  in  equation  (2)  in  order  to  ensure  that  our  results  are 
consistent  across  both  loss  functions.  Further  work  could  be  done  involving  more  general 
loss  functions,  or  calibrating  the  loss  functions  to  reflect  the  decision-maker’s  assessment 
of  relative  risk,  but  that  is  beyond  the  scope  of  this  study. 
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D.  FACTORS  AND  SETTINGS 


The  next  step  in  the  design  process  is  to  identify  factors  that  might  contribute  to 
the  MOE.  The  factors  are  classified  as  decision  factors  and  noise  factors. 

Decision  factors  are  controllable  in  the  real-world  setting  that  is  being  modeled. 
The  noise  factors  are  those  factors  that  are  not  controllable,  or  are  difficult  or  costly  to 
control.  Because  the  primary  goal  of  this  thesis  is  focused  on  the  VTUAV,  those 
decision  factors  associated  with  the  VTUAV  will  be  of  primary  interest.  Based  on  some 
exploratory  runs  in  MANA,  we  chose  to  use  the  seven  decision  factors  and  three  noise 
factors  in  Table  2  in  the  final  production  runs. 


DECISION  FACTORS 

Squad  Name 

State 

Factor  Name 

Low 

Setting 

High  Setting 

VTUAV 

Default 

Sweep  Width 

20 

250 

Observation  Cover 

0 

20 

Speed 

300 

1000 

Fatency 

0 

100 

Comm  Reliability 

5 

95 

Information  Accuracy 

5 

95 

Blue  Platoon 

Default 

Aggressiveness 

-5 

30 

NOISE  FACTORS 

Squad  Name 

State 

Factor  Name 

Low 

Setting 

High  Setting 

Civilian 

Default 

Red  Influence 

10 

50 

Red 

Enemy  Contact 

Aggressiveness 

-5 

30 

Default 

Stealth 

0 

90 

Table  2.  Decision  Factors  and  Noise  Factors 


In  Table  2,  the  decision  factors  and  noise  factors  are  outlined  by  squad  name,  state 
of  the  agent,  the  factor  name,  and  the  low  and  high  settings.  The  sweep  width  of  the 
VTUAV  is  related  to  the  field  of  view.  Because  the  battle  field  in  MANA  has  been 
defined  as  1  meter  equal  to  1  pixel,  the  conversion  is  straightforward  for  the  sweep  width. 
The  settings  range  from  20  to  250  meters.  The  observation  cover  factor  is  the  VTUAV’s 
propensity  to  provide  observation  cover  to  the  blue  platoon  upon  enemy  contact.  Speed 
in  MANA  is  relative  to  the  slowest  entity  on  the  battlefield.  In  our  scenario  the  red  and 
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blue  infantry  both  have  a  speed  of  5  mph.  The  VTUAV’s  speed  ranges  between  13 
nautical  miles  per  hour  and  44  nautical  miles  per  hour.  These  speeds  are  relatively  slow, 
but  consistent  with  urban  surveillance.  Latency  is  the  delay  in  communication  between 
the  VTUAV  and  the  blue  platoon.  The  settings  are  based  on  time  steps.  For  example,  a 
value  of  100  translates  into  a  100  time  step  delay  in  information  reaching  the  blue 
platoon.  Communication  reliability  is  the  likelihood  that  a  message  will  be  successfully 
delivered  to  the  receiving  squad.  Information  accuracy  is  the  probability  that  a  contact’s 
type  will  be  passed  correctly  to  the  receiving  squad.  The  final  factor  is  blue 
aggressiveness.  This  controls  the  blue  platoon’s  propensity  to  move  toward  enemy 
contacts. 

The  noise  factors  chosen  are  red  influence  on  the  civilians,  which  affects  civilian 
hostility  towards  blue  forces,  red  aggressiveness,  and  red  stealth.  A  civilian’s  hostility  is 
affected  by  nearby  red  agents.  Upon  contact  with  a  red  agent,  a  civilian  becomes  hostile 
towards  blue  forces  for  a  number  of  time  steps,  and  then  reverts  back  to  their  neutral 
state.  This  noise  factor  has  the  potential  to  be  somewhat  controllable  through  various 
pacification  efforts  not  explored  in  this  study.  For  the  purposes  of  our  investigation  we 
use  a  range  of  potential  hostility  to  include  civilian  hostility  as  a  noise  factor.  Red 
aggressiveness  and  stealth  are  the  final  two  noise  factors  for  our  study. 

E.  EXPERIMENTAL  DESIGN 

After  choosing  the  factors  and  levels  we  must  now  focus  on  the  data  collection 
plan.  For  our  study  we  will  use  a  crossed  decision  factor  x  noise  factor  plan.  This 
approach  uses  the  same  plan  for  each  of  the  noise  factors,  which  allows  us  to  examine  the 
variability  across  the  noise  space  [Sanchez  et  ah,  1996]. 

Utilizing  a  full  factorial  design  for  our  study  would  not  be  practical  given  the 
levels  we  wish  to  examine  and  expected  non-linear  response.  For  example,  if  we  were  to 
look  at  our  10  factors  each  having  only  4  levels,  we  would  need  410  design  points.  This 
would  require  1,048,576  runs  to  obtain  a  single  piece  of  data  for  each  design  point.  If  it 
took  only  one  minute  to  run  the  simulation  it  would  require  almost  2  years  for  a  single 
data  point  per  design  point.  Replicating  the  design  to  gather  infonnation  about  the 
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distribution  of  results  for  each  design  point  would  further  increase  the  time  required  for 
data  collection.  Obviously  this  approach  is  not  practical  for  even  a  modest  number  of 
factors  with  many  levels. 

Given  the  factors  and  levels  we  wish  to  examine,  we  chose  a  Latin  Hypercube 
Design  (LHC).  The  LHC  designs  are  excellent  for  space  filling  and  used  to  efficiently 
look  at  many  factors  with  many  levels  when  concerned  about  extreme  non-linearity 
[Lucas  et  ah,  2002].  This  is  done  by  not  only  sampling  on  the  edges  but  also  in  the 
interior  of  the  design  area.  For  our  study  we  will  use  a  Nearly  Orthogonal  LHC 
(NOLHC)  design.  This  design  provides  us  with  two  important  characteristics: 
orthogonality  and  good  space  filling  [Cioppa,  2002].  Orthogonality  is  important  in  order 
to  have  low  correlation,  reducing  the  effects  of  multicollinearity  on  the  model.  Good 
space-filling  is  important  because  the  design  points  are  scattered  throughout  the 
experimental  region  making  it  easier  to  detect  any  nonlinearities.  In  order  to  construct  a 
NOLHC  for  this  study,  a  Microsoft  Excel  Spreadsheet  (implementing  Cioppa’s  designs) 
constructed  by  Professor  Susan  Sanchez  was  used  to  generate  the  design  points  [Sanchez, 
2004].  Figure  1 1  displays  the  spreadsheet  for  a  1-7  factor  design.  The  user  can  pick  an 
orthogonal  or  nearly  orthogonal  design  based  on  the  number  of  factors  to  be  examined. 
The  minimum  and  maximum  levels  and  names  for  each  factor  are  then  entered  into  the 
appropriate  cells.  The  default  factor  levels  (given  as  the  integers  1-17  in  Figure  11)  are 
appropriately  converted.  Each  row  then  specifies  the  combination  of  factor  levels  for  a 
single  design  point.  The  ranges  used  in  our  investigation  mean  that  some  rounding  of  the 
factor  levels  occurs,  so  the  resulting  design  is  not  quite  orthogonal. 


32 


Figure  1 1 .  LHC  Design  Spreadsheet  (from:  Sanchez,  2004). 


In  order  to  better  utilize  the  space  filling  properties  of  the  LHC  design,  we  linked 
two  7  factor  OLH  designs  together  using  an  algorithm  described  by  Cioppa  [2002,  p.  54]. 
The  resulting  scatter  plot  and  correlations  for  each  pair  of  factors  are  shown  in  Figures  12 
and  13  respectively.  Each  region  is  reasonably  filled  with  points,  so  our  experiment  will 
provide  insight  for  intermediate  values  of  the  factors,  not  just  the  extremes.  When  linked 
together,  we  have  an  experimental  design  with  33  configurations.  Crossing  this  design 
with  a  similar  design  for  the  noise  factors  gives  us  1089  design  points  (33  x  33).  For 
each  of  the  design  points  we  will  conduct  40  replications  for  a  total  of  43,560  runs. 
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Figure  12.  Scatter  Plot. 

Figure  12  shows  that  our  design  has  reasonable  space-filling  behavior.  Table  3 
shows  that  despite  the  rounding  necessary  to  obtain  the  factor  levels  for  our  experiment, 
all  pairwise  correlations  are  still  very  near  zero. 


UAV 

Sweep 

Width 

Observation 

Cover 

UAV 

Speed 

UAV 

Latency 

UAV 

Conun 

Reliability 

UAV 

Info  Blue 

Accuracy  Aggressiveness 

Sweep  W"idth 

1 

-0.0007 

-0 

-0.0017 

0.0019 

o r 

-0.0086 

Observation  Cover 

-0.0007 

1 

0 

-0.0079 

0.009 

-0.0018 

0.0044 

UAV  Speed 

-0.0014 

0 

1 

0.0007 

0.0033 

oT~ 

0.0069 

UAV  Latency 

-0.0017 

-0.0079 

7E-04 

1 

0.0031 

0.0024 

-0.0038 

UAV  CommReliability 

0.0019 

0.009 

0.003 

0.0031 

1 

-0.0013 

-0.0045 

UAV  InfoAccuracy 

0 

-0.0018 

0 

0.0024 

-0.0013 

1 

0.0059 

BlueAggresiveness 

-0.0086 

0.0044 

0.007 

-0.0038 

-0.0045 

0.0059 

1 

Table  3.  Correlation  Matrix 
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F.  CONDUCT  THE  EXPERIMENT 

With  the  scenario  complete  and  design  points  determined,  the  factors  are  now 
ready  to  be  farmed.  The  MITRE  Corporation,  in  support  of  the  Marine  Corps’  Project 
Albert,  maintains  a  supercomputing  cluster  that  supports  data  fanning  projects.  After 
emailing  the  MITRE  support  team  an  XML  file  containing  the  base  scenario,  a  separate 
design  file  listing  the  factors  to  be  examined  and  the  conesponding  factor  settings  for 
each  design  point,  and  the  number  of  replications,  a  script  file  is  created  that  varies  the 
factor  levels  according  to  the  user-specified  design.  Output  files  are  returned  in  comma 
separated  value  (.csv)  format  [Hakola,  2004], 

After  running  40  replications  of  the  experiment  for  each  design  point,  we  collect 
the  mean  and  variance  for  each  point.  Letting  Y  be  the  random  function  of  the  decision 
factors  {X,}  and  the  noise  factors  {W.} ,  we  have  the  following  equations  for  the  mean 
response  Y.  and  mean  variance  V{  corresponding  to  the  decision  factor  configuration  i 
and  noise  configuration  j  : 


Y,  = 


i  ”«■  _ 


M 


(4) 


V,  =■ 


T£(7#-y,V+-£ 

1  M  nw  M 


(5) 


where  nw  is  the  number  of  noise  factor  points  [Sanchez  et  ah,  1996].  The  total  variance 

of  the  design  point  in  equation  (2)  is  the  sum  of  the  inherent  variance  of  the  simulation 
run  and  the  extrinsic  variance  across  the  noise  space. 

G.  ANALYZE  THE  RESULTS 

After  running  the  model  and  collecting  the  mean  and  variance  for  each  design 
point,  we  use  regression  to  develop  metamodels  for  the  mean  and  logarithm  of  the 
variance.  Regression  analysis  is  used  to  take  advantage  of  the  relationship  between  two 
or  more  variables  so  that  we  can  gain  infonnation  about  one  of  the  variables  by  knowing 
the  values  of  the  others  [Devore,  2000].  Our  metamodels  are  functions  of  the  decision 
factors  taken  over  the  noise  space.  We  use  a  design  for  fitting  main  effects,  interactions, 
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and  quadratic  effects.  The  variance  is  transformed  logarithmically  for  stability,  and  the 
initial  models  will  be  of  the  form  [Sanchez  et  ah,  1996]: 


Ll  ~  Y.  —  fio  +  fixXx  +  ■  ■  ■  +  /3kXk  +  fix2XxX2  +  ■  ■  ■  +  ftkXkXk_xXk  +  quadratic  (6) 
ln(<T‘)  ~  ln(F)  -yo  +  %Xx  +  ■  ■  ■  +  ykXk  +  y12X xX2  +  ■  ■  ■  +  fk_]kXk_]Xk  +  quadratic  (7) 


The  data  analysis  software  package  used  for  this  thesis  was  JMP  Statistical  Discovery 
Software™. 


H.  REFINE  THE  METAMODELS 

The  initial  models  will  be  evaluated  based  on  their  coefficient  of  detennination 
( R2 )  value.  This  value  is  the  proportion  of  variability  in  the  response  that  is  explained  by 
the  model.  Clearly,  R2  alone  may  not  be  the  best  measure  of  a  ‘good’  model.  With  a 
full  model  of  main  effects,  interactions,  and  second  order  polynomials  it  is  possible  to 
have  an  extremely  high  R2  but  few  degrees  of  freedom  for  error.  It  would  also  be 
difficult  to  explain  what  is  going  on  with  a  model  of  this  many  terms,  rendering  it 
virtually  useless.  The  initial  model  should  be  assessed  and  any  insignificant  terms 
removed.  The  test  used  to  detennine  which  individual  terms  will  remain  in  the  model  is 
the  t-test.  The  equation  for  the  t-statistic  is  the  following  [Hamilton,  1992]: 


t  = 


K~Pk 

SE, 


(8) 


where  bk  is  the  coefficient  estimate  for  factor  k ,  j8k  is  the  hypothesized  value  of  the 
coefficient  for  factor  k  ,  and  SEb  is  the  standard  error  of  bk  .  The  associated  /;- value  of 

the  t-statistic  will  determine  the  significance  of  the  term.  Reasonable  levels  of 
significance  are  usually  defined  as  any  /7-value  less  than  0.05  or  0.01.  For  our  study  we 
will  use  a  value  of  0.05. 

I.  SELECT  THE  ‘BEST’  DESIGN 

We  can  use  the  loss  function  to  examine  the  metamodels  to  determine  the  most 
desirable  decision  factor  values  for  our  scenario.  These  values  may  be  different  than 
those  we  would  suggest  if  we  were  basing  our  decision  on  mean  performance  alone. 
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This  chapter  has  outlined  the  robust  design  methodology  and  techniques  for 
analyzing  the  data  of  our  scenario.  Chapter  IV  will  present  a  detailed  analysis  based  on 
this  methodology. 
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IV.  DATA  ANALYSIS 


This  chapter  details  the  analysis  and  findings.  We  proceed  using  the  methodology 
described  in  the  previous  chapter.  We  will  begin  by  examining  the  data  and  giving  an 
initial  assessment  with  graphical  analysis  and  regression  trees.  We  will  then  analyze  the 
data  using  multiple  regression  models  with  main,  interaction,  and  quadratic  terms.  Next, 
we  will  refine  the  models  by  including  only  the  relevant  terms.  Using  the  refined  models 
with  the  loss  equation  detailed  in  Chapter  III,  we  will  select  the  best  design. 

A.  INITIAL  ASSESSMENT 

Upon  receipt  of  the  data,  we  examined  the  data  to  make  certain  there  were  no 
problems  with  the  data  quality.  We  consolidated  the  runs  of  each  of  the  1089  design 
points  and  produced  our  mean  exchange  ratios.  The  distribution  plot  of  the  data  is  shown 
in  Figure  13. 


Mean(Exhange  Ratio) 


- - ■> 

Moments 

Mean  0.9082214 

StdDev  0  1 121232 

Std  Err  Mean  0.0033977 
upper  95%  Mean  0.9148882 
lower  95%  Mean  0.9015547 
N  1089 


Figure  13.  Distribution  Plot  of  Mean(Exchange  Ratio). 

We  see  from  the  distribution  plot,  as  expected,  all  the  exchange  ratios  are  positive. 
Also  as  mentioned  in  Chapter  III,  all  mean  exchange  ratios  are  less  than  our  target  value 
of  1.5,  so  using  a  target  value  of  1.5  provides  a  motivation  toward  continual 
improvement.  The  distribution  has  positive  skewness,  including  some  high  values  that 

would  be  considered  outliers  if  the  data  followed  a  normal  distribution.  These  outliers 
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correspond  to  good  outcomes,  but  keep  in  mind  that  in  a  robust  design  framework  the 
best  decision  might  not  produce  the  same  outcome  as  a  decision  based  on  mean 
performance  alone. 

After  examining  the  data  we  then  developed  a  regression  tree,  which  can  give 
insights  to  the  possible  important  decision  factors  and  noise  factors  that  drive  our  model. 
A  regression  tree  partitions  the  data  recursively  according  to  the  relationship  between  the 
decision  factors  and  the  response  [JMP®  Statistics  and  Graphics  Guide,  2002],  but  does 
not  assume  a  specific  structure  for  the  form  of  this  relationship.  Because  the  focus  of  this 
study  is  developing  a  robust  model,  we  will  begin  by  developing  a  regression  tree  using 
only  the  decision  factors.  Later  in  the  chapter,  we  will  build  a  regression  tree  using  both 
the  decision  and  noise  factors  to  see  if  we  can  gain  additional  insights  into  the 
development  of  the  appropriate  tactics  in  various  settings.  Figure  14  is  a  regression  tree 
considering  only  the  decision  factors. 
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Figure  14.  Regression  Tree,  Decision  Factors  Only. 


With  33  design  points  represented  and  the  total  mean  shown  in  the  top  box,  we 

have  an  R2  of  .89  after  only  4  splits  of  the  data.  The  box  at  the  bottom  left  of  the  tree 

(colored  red)  represents  the  least  favorable  conditions,  yielding  an  average  exchange  ratio 

of  0.85.  The  box  at  the  bottom  right  of  the  tree  (colored  green)  represents  more  favorable 
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conditions,  yielding  an  average  exchange  ratio  of  0.99.  (In  this  instance  we  show  only 
one  red  and  one  green  leaf,  but  in  general  there  could  be  several  leaves  with  similar 
values.)  For  instance,  and  not  surprisingly,  the  best  exchange  ratios  are  achieved  with 
high  communication  reliability,  high  accuracy,  and  a  high  sweep  width.  It  is  interesting 
to  note  that  if  the  sweep  width  is  low  a  higher  accuracy  does  not  improve  the  mean. 

With  these  initial  insights  into  the  data  we  will  now  begin  to  develop  our 
regression  models  for  a  more  detailed  look  at  our  decision  factors. 

B.  DEVELOPMENT  OF  THE  MODELS 

As  outlined  in  the  previous  chapter,  we  use  regression  to  develop  metamodels  for 
the  mean  exchange  ratio  and  the  logarithm  of  the  variance  as  functions  of  the  decision 
factors. 

1.  Mean  Exchange  Ratio  Main  Effects 

The  regression  model  that  we  are  developing  associates  the  average  response 
(calculated  over  the  noise  space)  to  the  decision  factors.  We  began  by  using  only  main 
effects  to  get  an  idea  of  the  impact  of  these  factors  alone  on  our  model.  We  developed 
the  main-effects  model  using  the  mixed  stepwise  function  in  JMP.  This  function  uses 
alternating  forward  and  backward  steps,  allowing  terms  to  enter  the  model  on  the  forward 
step  and  leave  the  model  on  the  backward  step,  based  on  a  significance  level  for  each. 
For  our  model  we  allowed  terms  to  enter  the  model  at  a  significance  of  .25  and  removed 
tenns  with  significance  less  than  .05. 

The  main-effects  model  that  resulted  consisted  of  three  terms  with  an  R2  =  .78, 
indicating  that  78%  of  the  variability  in  the  exchange  ratio  is  explained  by  these  three 
tenns  (p-value  <  .0001).  Reliability,  accuracy,  and  aggressiveness  were  found  to  be 
significant  at  the  .05  level  and  the  plot  of  the  residuals  vs.  the  predicted  values  is  shown 
in  Figure  15.  The  solid  red  line  corresponds  to  predictions  from  the  regression  equation, 
while  the  dashed  red  lines  represent  a  95%  confidence  band  for  the  mean  predicted 
values.  Ideally,  the  points  should  be  scattered  around  the  prediction  line  with  no  apparent 
pattern.  We  see  from  the  graph  that  there  is  a  group  of  points  that  fall  below  the  lower 
confidence  limit.  While  the  R2  and  the  graph  look  reasonably  good,  those  points  indicate 
that  we  might  be  able  to  do  better. 
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Figure  15.  Actual  vs.  Predicted  Exchange  Ratio. 

2.  Final  Model  Mean  Exchange  Ratio 

Again  using  the  stepwise  function  in  JMP  with  the  same  significance  criteria  in 
the  main  effects  model,  we  developed  the  final  model.  The  final  model  consisted  of  six 
terms:  four  main  effects,  one  interaction,  and  one  quadratic  with  an  overall  p-value  < 
.0001.  Looking  at  the  actual  versus  predicted  exchange  ratio  in  Figure  16  we  see  that  the 
group  of  troublesome  points  in  Figure  15  has  decreased.  There  does  not  appear  to  be  any 
pattern  in  this  plot,  indicating  the  assumption  of  independent  and  identically  distributed 
errors  is  reasonable.  The  R*  2  for  this  model  is  .91,  giving  us  better  predictability  than  just 
the  main-effects  model  while  still  having  relatively  few  terms.  The  final  model  of  the 
exchange  ratio  is  represented  by  the  following  equation: 

ExhangeRatio  ~  0. 7 598  +  0. 00007 (Sweep  _Width)+ 0.00157 (Reliability) 

+0.00 1 22i  Accuracy) + 0.00 149 (Blue  _  Aggressiveness ) 
-A).000022[(Reliability-50.0606)*(Acci(racy-50.0606)] 

-0.0000( )4[(  Sweep  _  Width  -135.061)  *  {Sweep _  Width  - 135.061)]  (8) 

The  interaction  and  quadratic  term  have  the  mean  point  subtracted  from  the  factor  value 
to  ensure  proper  scaling. 


80  .85  .90  .95  1.00  1.05 

Exhange  Ratio  Predicted  P<.0001 
RSq=0.78  RMSE=0.0323 
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Mean(Exhange  Ratio)  Predicted  P<.0001 
RSq=0.91  RMSE=0.0218 
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Figure  16.  Actual  and  Residual  vs.  Predicted  Exchange  Ratio. 


Summary  statistics  for  the  final  model  can  be  seen  in  the  Appendix.  Of  the  seven 
decision  factors,  four  appear  as  main  effects  in  the  final  model.  Although  sweep  width 
has  a  p-value  of  .19  it  has  been  included  as  a  main  effect  due  its  significance  (p-value  < 
.0002)  as  a  quadratic  term  in  the  model.  Figure  17  displays  the  leverage  plots  for  the 
significant  main  effects  of  the  model.  The  UAV  communications  reliability  has  the 
strongest  influence,  followed  by  UAV  information  accuracy  and  blue  aggressiveness, 
respectively. 
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Figure  17.  Leverage  Plots. 


The  model’s  interaction  tenn,  communication  reliability  and  information 
accuracy,  can  be  seen  nicely  using  the  contour  plot  in  Figure  18.  The  blue  regions  circled 
in  the  upper  right-hand  corner  represent  areas  with  higher  mean  exchange  ratios. 
Intuitively  it  seems  reasonable  that  these  two  factors  would  have  an  interaction  effect. 
Perfect  communication  with  imperfect  information  or  vice  versa  is  virtually  useless, 
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but — like  the  regression  tree — the  contour  plot  can  give  us  insights  into  the  potential 
trade  space  between  these  two  factors.  This  can  be  especially  important  in  an  urban 
environment  where  potential  threats  can  be  difficult  to  identify  and  communication 
sporadic. 
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Figure  18.  Contour  Plot  of  Accuracy  and  Reliability  vs.  Mean  Exchange  Ratio. 

The  contour  plot  in  Figure  19  shows  the  mean  exchange  ratio  as  a  function  of  blue 
aggressiveness  vs.  information  accuracy.  As  the  circled  area  on  this  plot  shows,  we  see 
that  as  the  information  we  receive  becomes  more  accurate  the  need  for  higher 
aggressiveness  in  order  to  maintain  high  exchange  ratios  decreases.  This  could  have 
potential  benefits  in  areas  where  we  want  to  minimize  our  aggressive  posture  in  order  to 
win  the  hearts  and  minds  of  the  local  population. 
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Figure  19. 


Contour  Plot  of  Accuracy  and  Aggressiveness  vs.  Mean  Exchange  Ratio. 


The  final  term  in  the  model  is  the  quadratic  tenn  of  sweep  width.  This  term 
implies  that  there  is  a  point  where  too  large  of  a  field  of  view  is  detrimental  to  the 
exchange  ratio.  Generally  this  would  seem  counterintuitive;  the  wider  area  a  UAV  can 
see  the  better.  However,  if  we  look  at  this  factor  in  the  context  of  the  situation  we  are 
modeling  it  seems  to  make  more  sense.  The  area  of  the  patrol  is  relatively  small  and  the 
capability  of  a  small  foot  mobile  unit  to  cover  large  distances  in  an  urban  environment  is 
limited.  With  the  UAV  able  to  detect  more  red  agents  at  greater  distances,  the  blue  forces 
will  move  to  engage  these  detections.  If  the  detections  are  lost  by  the  UAV  before  the 
blue  forces  arrive,  the  blue  forces  will  move  back  to  their  original  patrol  route  or  to  the 
most  current  UAV  detection.  Moving  over  larger  distances  it  is  not  difficult  to  imagine 
an  accordion  effect  between  more  recent  detections,  resulting  in  fewer  engagements  and 
lower  exchange  ratios.  It  might  be  argued  that  this  is  an  artificiality  of  the  model,  but  it 
could  also  be  seen  as  information  overload.  With  too  much  information  the  blue  force 
becomes  indecisive  and  ineffective.  Figure  20  displays  the  prediction  profiler  graph  from 
JMP.  The  profiler  is  a  useful  tool  to  see  how  the  predicted  values  change  as  you  change 
one  factor  at  a  time.  Figure  shows  the  settings  that  maximize  the  mean  exchange  ratio 
based  on  the  metamodel.  Notice  the  curve  for  sweep  width  and  the  point  of  decreasing 
effectiveness.  This  means  that  moderate  sweep  values  are  best. 
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Figure  20.  Prediction  Profiler  with  Optimal  Settings  for  Maximum  Exchange  Ratio. 


3.  ln(Variance)  Model 

The  ln(Variance)  model  was  developed  in  the  same  manner  as  the  mean  exchange 
ratio  model.  We  considered  all  main  effects,  two-way  interactions,  and  quadratic  effects, 
allowing  terms  to  enter  the  model  with  significance  greater  than  0.25  but  eventually 
eliminating  terms  with  significance  less  than  .05.  Using  the  stepwise  function  in  JMP  we 
obtain  the  following  metamodel  equation: 

ln(<72)  ~  -2.500952  +  0.00060(SVreep _Width)  +  0.0067 {(Reliability) 

+0.00532 (Accuracy)  +  0.00619 {Blue  _  Aggressiveness) 

+0.00003[(SWep  _  Width  - 135.06 1)  *  ( Accuracy  -  50.0606)]  (9) 

-0.00002 [(SweepWidth  - 135.061)  *  ( Sweep  _  Width  - 135.061)] 

-0.000 1 3[( Reliability  -  50.0606)  *  ( Reliability  -  50.0606)] 

Summary  statistics  for  the  final  model  of  ln(variance)  can  be  seen  in  the 
Appendix.  Figure  21  displays  the  graphs  for  actual  and  residual  by  predicted.  We  see 
from  the  graph  that  the  model  produces  an  R2  of  .85.  The  regression  is  statistically 
significant  (p-value  <  .0001). 
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Figure  21.  Actual  and  Residual  vs.  ln(Variance)  Predicted. 


In  order  to  make  use  of  the  maximization  function  in  the  profiler  we  maximized 
the  quantity  [-ln(variance)]  in  order  to  get  the  best  settings  for  minimum  variance  as  seen 
in  Figure  23. 


Figure  22.  Prediction  Profiler  with  Optimal  Settings  for  Minimum  Variance. 

Looking  at  the  optimal  settings  for  minimum  variance,  the  values  are  almost  all  at  the 
other  end  of  the  spectrum  from  the  optimal  settings  for  the  maximum  exchange  ratio. 
With  the  settings  for  reliability  and  accuracy  at  5  the  UAV  is  virtually  nonexistent  as  an 
intelligence  gathering  asset.  The  aggressiveness  setting  at  -5  translates  into  a  blue  force 
avoiding  engagements  with  the  red  forces.  Essentially,  in  order  to  keep  the  variance  at  a 
minimum,  the  blue  force  should  avoid  fighting.  Of  course  this  is  not  an  option  in  urban 
combat. 
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In  an  ideal  world  we  would  be  able  to  simultaneously  maximize  the  exchange 
ratio  and  minimize  the  variance  using  the  same  decision  factor  settings.  Unfortunately 
this  is  rarely  the  case.  As  the  profiler  plots  and  Figure  23  demonstrate,  for  our  scenario  it 
is  impossible.  Figure  23  displays  the  mean  exchange  ratio  vs.  the  variance.  We  see  that 
as  the  mean  exchange  ratio  increases  so  (in  general)  does  the  variance.  Even  though 
there  does  appear  to  be  a  general  trend,  this  is  not  a  direct  relationship.  There  are 
differences  in  the  variance  especially  at  high  exchange  ratios.  This  is  not  surprising 
given  that  in  order  to  increase  the  number  of  red  kills,  the  blue  forces  will  have  to  engage. 


Figure  23.  Variance  vs.  Mean  (Exchange  Ratio). 

We  have  seen  from  the  profiler  that  the  best  mean  exchange  ratio  the  model  will 
achieve  is  1.16;  therefore,  if  we  are  to  have  a  chance  to  come  close  to  our  target  value  of 
1.5  we  will  need  increased  variability  in  the  loss  function.  (Note  that  the  increased 
variability  also  means  the  exchange  ratio  may  be  much  lower.)  Now  that  we  have 
developed  the  metamodels  for  the  mean  exchange  ratio  and  the  variance  of  the  exchange 
ratio,  we  will  now  use  equations  (8)  and  (9)  in  order  to  minimize  the  loss  with  respect  to 
our  target  value  of  1 .5. 
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C.  CHOOSING  THE  ‘BEST’  DESIGN 

Using  equations  (8)  and  (9),  along  with  the  quadratic  loss  equation  (2),  we  can 
look  for  the  most  robust  configuration.  Because  the  data  collection  plan  uses  a  LHC 
design  rather  than  a  full  factorial  we  will  use  a  grid  search  over  the  model’s  decision 
factors  for  the  minimum  loss. 

1.  Minimizing  the  Loss  Function 

In  order  to  look  at  all  possible  combinations  of  all  levels  of  the  four  decision 
factors  common  to  both  metamodels  would  require  65,205,000  combinations.  Instead  of 
looking  at  each  level  we  aggregate  the  intervals  to  a  more  manageable  amount  of 
combinations.  Using  the  factors  and  levels  of  Table  4,  we  are  able  to  look  at  a  grid  of 
9,600  combinations. 
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65 

65 
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85 

200 

95 

95 

220 

240 

Table  4.  Loss  Factors  and  Levels 


Table  5  contains  the  33  design  points  with  the  resulting  mean,  ln(variance)  and 
loss  for  our  target  value  of  1.5.  The  design  point  with  the  lowest  loss  tested  is  the  7th  row 
(boldfaced  and  highlighted  in  yellow)  with  a  loss  of  0.433.  The  design  point  in  row  3 
yields  the  lowest  (worst)  mean  exchange  ratio,  but  the  best  (lowest)  ln(variance);  its  loss 
is  0.547 — 26%  higher  than  the  most  robust  design  point.  The  losses  associated  with  the 
design  points  corresponding  to  the  best  mean  exchange  ratio  (1.09)  and  the  worst 
ln(variance)  (-1.20)  are  .436  and  .534,  respectively. 
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33 
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Table  5.  Mean,  ln(variance),  and  Loss(1.5)  for  the  33  Design  Points 


Using  equation  (2)  we  determined  the  factor  settings  associated  with  minimum 
loss  for  the  target  value  of  1 .5  was  a  combination  that  had  not  been  tested:  Sweep  Width 
=  20,  Accuracy  =  95,  Reliability  =  95,  and  Aggressiveness  =  30.  The  loss  associated  with 
these  settings  was  .343  which  was  21%  less  than  the  best  of  the  design  points  tested.  A 
validation  experiment  was  run  at  this  setting  to  see  whether  or  not  the  final  metamodels 
are  still  providing  a  good  fit  for  these  factor  settings.  The  validation  experiment  yielded  a 
mean  exchange  ratio  of  1.06,  which  is  within  the  95%  prediction  interval  of  (1.007, 
1.123).  Similarly  the  variance  and  loss  for  the  validation  experiment,  which  were  .149 
and  .339  respectively,  fell  within  the  95%  prediction  intervals  computed  from  the 
metamodels.  This  is  strong  evidence  that  the  model  developed  from  only  33  design 
points  is  providing  good  results  in  other  parts  of  the  design  space.  This  is  even  more 
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impressive  if  we  consider  that  the  smallest  full  factorial  design  that  could  have  looked  for 
main,  quadratic  and  interaction  effects  would  have  needed  2187  design  points  for  the 
decision  factors.  A  full  factorial  with  33  levels  for  each  decision  factor  would  require 
over  42.6  billion  design  points. 

2.  Insights  into  the  ‘Best’  Design 

With  the  best  design  for  our  model  chosen,  we  then  decided  to  examine  the  loss 
function  over  a  large  range  of  possible  targets.  As  mentioned  earlier,  exchange  ratios 
have  varied  a  great  deal  in  recent  history.  We  decided  to  look  at  target  values  ranging 
from  1  to  2  with  increments  of  .01  for  a  total  of  101  different  target  values.  Taking  the 
minimum  loss  value  for  each  target  yielded  the  twelve  different  designs  (or  factor  setting 
combinations)  shown  in  Table  6. 


Factor  Setting 

Sweep  Width 

Accuracy 

Reliability 

Aggressiveness 

1 

240 

5 

5 

30 

2 

20 

95 

95 

-5 

3 

20 

95 

95 

0 

4 

20 

95 

95 

5 

5 

20 

95 

95 

10 

6 

20 

95 

95 

15 

7 

20 

95 

95 

20 

8 

20 

95 

95 

25 

9 

20 

95 

95 

30 

10 

40 

95 

95 

30 

11 

60 

95 

95 

30 

12 

80 

95 

95 

30 

Table  6.  Factor  Settings  for  Minimum  Loss  for  Target  Values  1  Through  2 

Figure  24  shows  the  best  factor  settings  for  a  particular  target.  Notice  that  for  a 
target  value  anywhere  from  1.37  to  1.67,  factor  setting  9  produces  the  minimum  loss.  For 
targets  less  than  1.1  we  see  the  factor  settings  approach  those  that  produce  minimum 
variance.  Although  not  shown  in  the  Figure,  a  target  loss  of  .8  results  in  factor  settings 
that  are  equal  to  those  that  produce  minimum  variance.  Conversely,  as  target  values  rise 
above  1.1,  factor  settings  approach  those  that  maximize  the  mean. 
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As  target  values  rise  above  the  mean  of  the  data,  the  loss  function  favors  settings 
that  maximize  the  mean  and  variance.  In  order  to  reach  the  target  value  we  not  only  must 
try  to  maximize  the  exchange  ratio,  but  we  must  also  increase  the  variability  so  that  we 
have  the  potential  of  reaching  the  target.  As  stated  earlier,  in  a  perfect  world  we  want  the 
exchange  ratio  to  be  as  high  as  possible,  but  in  order  to  achieve  that  goal  we  must  be 
willing  to  accept  a  large  amount  of  risk. 

Recall  from  Chapter  III  that  in  an  ideal  situation  our  target  value  would  be  equal 
to  the  mean  and  our  variance  would  be  zero.  Figure  25  represents  that  situation  in  as 
much  as  the  current  model  can  produce  it.  With  our  target  value  at  .78  and  our  variance 
minimized,  we  see  the  lowest  loss  that  our  model  can  produce.  The  settings  for  this 
target  value  are:  Sweep  Width  =  240,  Accuracy  =  5,  Reliability  =  5  and  Aggressiveness  = 
-5.  These  settings  are  the  equivalent  to  not  fighting — which  is  obviously  not  a  choice  in 
our  scenario.  The  target  value  would  not  even  be  worthwhile  to  consider,  but  the  graph 
does  give  visual  insights  about  the  effect  of  the  target  value  chosen  and  the  importance  of 
variance  on  the  loss  function. 
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Let  us  examine  the  factors  in  Table  4  a  little  closer.  Notice  that  for  targets 
between  1  and  1.1,  the  UAV  is  essentially  turned  off  and  the  blue  forces  are  very 
aggressive.  At  values  below  the  target  of  1,  blue  forces  eventually  become  the  least 
aggressive.  While  these  settings  might  produce  the  lowest  loss  of  the  entire  model,  it  is 
akin  to  the  adage  of  “nothing  ventured,  nothing  gained.”  In  turn,  if  we  look  at  factor 
settings  2  through  9,  we  see  that  the  only  decision  factor  that  is  changing  is  the 
aggressiveness  of  the  blue  forces.  In  other  words,  the  only  change  in  factor  settings  that 
is  required  to  minimize  the  loss  for  target  values  1.1  through  1.68  is  the  blue  force’s 
aggressiveness.  Once  the  aggressiveness  setting  reaches  its  maximum  of  30,  the  sweep 
width  slowly  increases  to  a  maximum  of  80  for  the  target  value  of  2. 

D.  AN  ALTERNATE  LOSS  FUNCTION 

Now  let  us  consider  the  alternate  loss  function  of  equation  (3)  that  was  discussed 
in  Chapter  III.  By  calculating  the  loss  based  on  -log(exchange  ratio)  and  plotting  this 
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against  the  loss  of  the  quadratic  function  of  equation  (2)  we  can  determine  how  closely 
the  two  sets  of  results  correspond  to  each  other.  Figure  26  shows  a  scatter  plot.  With  an 
R2  of  0.74  and  a  correlation  of  0.93  the  alternate  loss  function  is  in  close  agreement  with 
the  quadratic  loss  using  a  target  of  1.5.  Two  design  points  might  be  considered  outliers, 
since  they  have  higher  quadratic  losses  than  others  with  similar  -log(exchange  ratio) 
values,  but  these  are  not  associated  with  the  most  robust  design  points. 
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Figure  26.  Loss  Function  Comparison 


E.  ROBUSTNESS  REVISITED 

Recall  from  the  beginning  of  the  chapter  that  we  developed  a  decision  tree  using 
only  the  decision  factors  taken  over  the  noise  space.  Now  let  us  construct  a  new  decision 
tree  using  all  of  the  decision  and  noise  factors  using  loss  as  our  response  instead  of  the 
exchange  ratio.  We  can  see  from  Figure  27  that  similar  splits  occur. 


54 


r  a 

All  Rows 
Count  =  1089 
Mean  Loss  =  0.50 


Civilian  Hostility  <  33 

Civilian  Hostility  >  33 

Count  =  528 

Count  =  561 

Mean  Loss  =  0.46 

Mean  Loss  =  0.54 

Accuracy  >  78 
Count  =  425 
Mean  Loss  =  0.43 


Accuracy  <  78 
Count  =  136 
Mean  Loss  =  0.47 


Comm  Reliability  >  61 
Count  =  304 
Mean  Loss  =  0.51 


Accuracy  >  39 
Count  =  160 
Mean  Loss  =  0.49 

V _ J 


Figure  27.  Regression  Tree,  All  Factors. 


The  regression  tree  in  Figure  27  represents  all  1089  design  points  and  considers 
noise  and  decision  factors  together.  The  overall  mean  is  shown  in  the  top  box  and  the 
tree  has  an  R2  of  .24.  The  boxes  colored  red  and  green  at  the  bottom  of  the  tree  represent 
the  less  and  more  favorable  conditions,  respectively.  For  instance,  according  to  the 
regression  tree,  if  we  were  in  a  situation  were  the  civilian  hostilities  were  high,  it  would 
be  desirable  that  our  communication  and  accuracy  also  be  high. 

The  biggest  factor  impacting  the  mean  loss  is  the  influence  on  the  civilian 
population  by  the  red  forces,  or  civilian  hostility.  When  the  factor  setting  is  greater  than 
33  the  mean  loss  is  .54,  if  it  is  less  than  33  the  loss  falls  to  .46.  Remember  the  influence 
ranged  between  10  and  50,  so  a  value  of  33  would  represent  an  area  with  just  above  50% 
hostile  civilians.  Depending  on  the  level  of  hostility  by  the  civilian  population, 
communication  reliability  and  information  accuracy  are  the  second  most  important 
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factors  on  the  tree.  If  the  patrol  is  operating  in  an  environment  with  high  civilian 
hostility,  greater  communication  reliability  and  infonnation  accuracy  are  both  needed  in 
order  to  avoid  high  loss.  Alternately  if  the  civilian  hostilities  are  low,  high  information 
accuracy  is  the  most  important  contributor  to  low  loss. 

The  regression  tree  also  gives  us  an  idea  of  the  potential  trade  space  given  that  we 
possess  certain  kinds  of  intelligence.  For  instance,  looking  at  the  regression  tree  we  see 
that  in  situations  where  the  local  populace  is  relatively  friendly  to  our  forces,  we  might  be 
able  to  achieve  acceptable  results  even  with  a  UAV  that  has  low  communication 
reliability  and  low  accuracy. 

F.  SUMMARY 

In  this  chapter  we  developed  models  for  the  mean  exchange  ratio,  as  well  as  the 
variance  of  the  exchange  ratio.  The  mean  exchange  ratio  model  consisted  of  seven  tenns 
that  explained  91%  of  the  variability,  with  three  of  these  terms  explaining  74%  of  the 
variability.  The  ln(variance)  model  consisted  of  8  terms  explaining  84%  of  the 
variability.  Using  these  two  models  and  the  loss  function,  we  developed  insights  into  the 
best  model  based  on  a  specific  as  well  as  a  range  of  target  values.  We  also  developed 
regression  trees  for  an  initial  and  nonparametric  understanding  of  our  data  set. 

Recall  that  the  goal  of  this  thesis  is  to  provide  insights,  rather  than  numbers,  so 
our  numerical  results  should  be  viewed  in  this  light.  The  final  chapter  will  link  the 
observations  from  the  statistical  analysis  with  possible  implications  for  the  real  world. 
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IV.  CONCLUSIONS  AND  RECOMMENDATIONS 


Recall  that  the  purpose  of  this  thesis  was  to  gain  insights  to  small  unit  urban 
patrolling  with  the  VTUAV  using  agent  based  simulation.  We  now  summarize  our 
findings  and  provide  recommendations  for  follow-on  research. 

A.  SUMMARY 

The  metamodels  and  designs  developed  for  the  agent  based  simulation  of  this 
thesis  do  not  translate  literally  into  the  real  world  of  modem  combat,  nor  is  this  the 
purpose  of  agent  based  distillations.  What  we  have  been  able  to  observe  using  these 
models  gives  us  insights  into  the  potential  tactics,  techniques,  and  capabilities  that  might 
be  important  in  similar  situations.  It  also  provides  insights  into  the  possible  trade  space 
of  technology  and  asset  support  given  the  constant  battle  over  limited  resources.  This 
thesis  also  provides  a  template  for  studying  other  scenarios  using  robust  design. 

As  stated  in  the  ORD  for  the  VTUAV,  it  must  be  able  to  accomplish  missions 
ranging  from  MOOTW  to  a  large  intensity  conflict.  Because  it  must  be  able  to  operate  in 
different  environments  at  different  levels  of  conflict,  the  VTUAV  cannot  be  specifically 
designed  for  just  one  type  of  operation.  Given  this  requirement,  this  study  focused  on  a 
scenario  with  a  broad  range  of  environmental  noise  factors  and  intensity  levels  in  order  to 
generate  a  robust  system  capable  of  operating  in  different  environments. 

A  platfonn  that  has  very  reliable  communication  capability  and  very  accurate 
sensing  capability  is  best  suited  for  the  environment  modeled.  These  two  factors  were 
the  most  dominant  in  providing  favorable  results.  A  UAV  with  high  reliability  and 
accuracy  needs  only  a  relatively  narrow  field  of  view  to  provide  favorable  results  for 
small  units  operating  in  dense  urban  environments.  Speed,  as  might  have  been  guessed, 
will  probably  not  play  an  important  role  for  a  UAV  in  an  urban  environment.  These 
factors  might  suggest  possible  flexibility  when  using  UAVs  in  support  of  urban 
operations.  Because  of  its  broad  range  of  required  capabilities,  the  VTUAV  will  be  a 
very  capable,  expensive  and  limited  asset.  If  other,  less  expensive  assets  can  be  used, 
cost  savings  and  flexibility  could  be  realized.  As  mentioned  in  the  introductory  chapter, 
the  Marine  Corps  currently  possesses  the  Dragon  Eye  UAV.  Although  a  slow  and  rather 
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unsophisticated  over-the-hill  reconnaissance  asset,  upgrading  its  sensing  capabilities 
might  make  the  Dragon  Eye  a  low  cost  alternative  to  using  the  VTUAV  in  the  urban 
environment. 

By  introducing  noise  into  the  system  and  using  the  tool  of  regression  trees,  we  are 
given  an  idea  about  the  potential  trade  space  of  required  capabilities  given  our  knowledge 
of  certain  conditions.  For  instance,  we  saw  that  the  hostility  of  the  civilian  population 
plays  an  important  role  in  urban  combat.  If  civilian  hostilities  are  high,  a  more 
sophisticated  UAV  appears  to  be  necessary  for  the  patrol  to  operate  effectively.  In 
environments  with  more  friendly  populations,  the  Dragon  Eye  or  other  less  sophisticated 
intelligence  assets  might  be  sufficient  to  accomplish  the  mission. 

We  see  from  the  metamodels  and  the  loss  function  analysis  that  the  intelligence 
gathered  from  the  UAV  is  only  a  piece  of  the  puzzle.  How  the  blue  force  uses  that 
intelligence  is  the  other  piece.  The  aggressiveness  of  the  blue  force  has  an  impact  on  the 
mean  exchange  ratio.  If  higher  ratios  are  to  be  achieved  in  this  model,  the  blue  force 
must  be  more  aggressive.  With  increased  aggressiveness  comes  higher  variance, 
resulting  in  higher  rewards  but  also  higher  risks.  In  situations  were  commanders  cannot 
tolerate  the  possibility  of  high  losses,  a  less  aggressive  force  that  relies  more  heavily  on 
the  UAV  for  situational  awareness  and  enemy  engagements  might  be  more  appropriate. 

B.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

Because  no  two  combat  situations  are  alike,  the  possible  variations  of  combat  that 
one  can  attempt  to  model  are  infinite.  This  analysis  was  meant  to  provide  insights,  but 
with  those  insights  comes  more  questions.  Listed  below  are  just  a  few  of  the  potential 
areas  of  future  research  stemming  from  this  scenario: 

•  This  model  uses  only  one  UAV  due  to  its  small  area  of  operations.  What 
would  be  the  impact  of  using  other  intelligence-gathering  devices,  such  as 
unmanned  ground  vehicles  and  fixed  position  sensors,  in  conjunction  with 
the  UAV? 

•  The  UAV  in  this  scenario  is  invulnerable  to  enemy  fire  and  is  assumed  to 
be  always  operational.  While  statistics  show  that  the  downing  of  UAVs  is 
rare,  the  slow  speeds  required  in  urban  environments  might  make  it  more 
likely  to  be  shot  down.  How  would  this  vulnerability  affect  the  results? 

•  What  would  be  the  effect  of  limiting  the  time  on  station  of  the  UAV  due  to 
endurance  limitations  or  competing  requirements  from  other  units? 
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•  This  model  assumes  fairly  strict  ROE  and  therefore  does  not  take  into 
account  the  combined  arms  capability  of  our  forces.  How  would  modeling 
this  capability  affect  the  outcome? 

•  This  model  looked  at  only  one  performance  measure,  the  exchange  ratio. 
Future  studies  might  include  other  performance  measures  involving 
civilian  casualties  or  time  to  complete  the  mission. 

•  Other  loss  functions  may  be  better  suited  for  combat  analysis.  Any  future 
work  using  robust  design  should  consider  using  a  loss  function  that  is  the 
most  appropriate  for  the  MOE  being  considered. 

With  a  little  imagination  the  list  of  variations  is  virtually  endless.  This  thesis  provides 

some  preliminary  insights  into  the  capabilities  the  VTUAV  should  possess.  Other 

questions  will  arise  over  the  next  few  years  as  the  VTUAV  is  developed  and  fielded. 

Once  its  capabilities  are  fully  known,  the  research  focus  can  turn  to  the  best  ways  to 

employ  this  new  asset. 
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APPENDIX.  FINAL  REGRESSION  MODELS  SUMMARY 

STATISTICS 


A.  MEAN  EXCHANGE  RATIO  MODEL 
Summary  of  Fit 

RSquare 
RSquare  Adj 
Root  Mean  Square  Error 
Mean  of  Response 
Observations  (or  Sum  Wgts) 


F  Ratio 
44.9725 
Prob  >  F 
<.0001 


Analysis  of  Variance 

Source 

DF 

Sum  of  Squares 

Mean  Square 

Model 

6 

0.12804066 

0.021340 

Error 

26 

0.01233737 

0.000475 

C.  Total 

32 

0.14037803 

0.912113 

0.891832 

0.021783 

0.908221 

33 


Parameter  Estimates 

Term 

Intercept 

UAV  Sweep  Width 
UAV  CommReliability 
UAV  InfoAccuracy 
BlueAggresiveness 

(UAV  CommReliability-50.0606)*(UAV  InfoAccuracy- 
50.0606) 

(UAV  Sweep  Width-135. 061)*(UAV  Sweep  Width- 
135.061) 


Estimate  Std  Error  t  Prob>|t| 
Ratio 

0.7597702  0.014026  54.17  <.0001 
0.0000713  0.000053  1.34  0.1904 

0.0015703  0.000136  11.57  <.0001 
0.0012232  0.000136  9.01  <.0001 

0.0014883  0.000348  4.28  0.0002 


0.0000218  0.000005  4.41  0.0002 


-0.000004  8.651e-7  -4.43  0.0001 


Effect  Tests 

Source 

UAV  Sweep  Width 
UAV  CommReliability 
UAV  InfoAccuracy 
BlueAggresiveness 

UAV  CommReliability*UAV  InfoAccuracy 
UAV  Sweep  Width*UAV  Sweep  Width 


Nparm  DF  Sum  of  Squares  F  Ratio  Prob  >  F 

1  1  0.00085761  1.8073  0.1904 

1  1  0.06349981  133.8206  <.0001 

1  1  0.03852984  81.1985  <.0001 

1  1  0.00868285  18.2984  0.0002 

1  1  0.00921669  19.4234  0.0002 

1  1  0.00933128  19.6649  0.0001 
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B.  LN(VARIANCE)  EXCHANGE  RATIO  MODEL 
Summary  of  Fit 


RSquare  0.848906 

RSquareAdj  0.8066 

Root  Mean  Square  Error  0.136172 

Mean  of  Response  -1.95967 

Observations  (or  Sum  Wgts)  33 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

7 

2.6045342 

0.372076 

20.0657 

Error 

25 

0.4635717 

0.018543 

Prob  >  F 

C.  Total 

32 

3.0681059 

<.0001 

Parameter  Estimates 

Term  Estimate  Std  Error  t  Prob>|t| 


Ratio 

Intercept 

-2.500952  0.09381 

-26.66 

<.0001 

UAV  Sweep  Width 

0.0005978  0.000331 

1.80 

0.0834 

UAV  CommReliability 

0.0067098  0.000849 

7.91 

<.0001 

UAV  InfoAccuracy 

0.0053155  0.000849 

6.26 

<.0001 

BlueAggresiveness 

0.0061833  0.002175 

2.84 

0.0088 

(UAV  Sweep  Width-135.061)*(UAV  InfoAccuracy- 
50.0606) 

0.0000303  0.000014 

2.13 

0.0433 

(UAV  Sweep  Width-135. 061)*(UAV  Sweep  Width- 

-0.000023  0.000005 

-4.25 

0.0003 

135.061) 

(UAV  CommReliability-50. 0606)*(UAV 

-0.00013  0.000041 

-3.16 

0.0041 

CommReliability-50. 0606) 


Effect  Tests 


Source 

Nparm  DF 

Sum  of  Squares 

F  Ratio  Prob  >  F 

UAV  Sweep  Width 

1 

1 

0.0603143 

3.2527 

0.0834 

UAV  CommReliability 

1 

1 

1.1594423 

62.5277 

<.0001 

UAV  InfoAccuracy 

1 

1 

0.7276172 

39.2397 

<.0001 

BlueAggresiveness 

1 

1 

0.1498617 

8.0819 

0.0088 

UAV  Sweep  Width*UAV  InfoAccuracy 

1 

1 

0.0840276 

4.5315 

0.0433 

UAV  Sweep  Width*UAV  Sweep  Width 

1 

1 

0.3347629 

18.0535 

0.0003 

UAV  CommReliability*UAV  CommReliability 

1 

1 

0.1846960 

9.9605 

0.0041 
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